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ABSTRACT 


A tool that measures the effectiveness of software project management can be 
used to identify strengths and weaknesses, and guide improvement to practices 
in order to increase the chances of project success. The Software Project 
Management Effectiveness (PME) Metric is one such tool that has shown 
promise in this area of software engineering. To discover how promising the 
metric is, nine software practitioners participated in this research and assisted 
with measuring projects they recently worked on. A strong correlation between 
the PME metric and project success was identified. The software practitioners 
also provided feedback on the usefulness and applicability of the PME metric. 
seventy-five percent of the software practitioners stated that they would use the 
metric on the next project they worked on. This research has found that the PME 
metric should be considered for use by project managers who continuously want 


to improve and deliver successful software projects. 
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I. INTRODUCTION 


A. INTRODUCTION 


The United States Department of Defense (DoD) recently reported that the 
development costs of 72 weapons programs had climbed 40 percent from their 
initial estimates, there was an average delay of 21 months, and the total systems 
overrun was $2 billion dollars (Government Accountability Office, 2008). Studies 
show that these development problems are typically not caused by technology 
issues but are largely due to program management (Office of the Under 
Secretary of Defense, 2000). Improving program management should be a 
primary focus of the DoD if there is to be any hope of significantly increasing 
program performance. One of the key aspects of the DoD’s program 


management is the management of system software development. 


Software has become such an integral part of weapon systems that it is 
virtually impossible to find a weapons system today that does not contain 
mission-critical software at its core (Welby, 2010). This is not just isolated to the 
DoD. Reliance on software keeps growing in industries as diverse as transport, 
medical, communications, energy, space, entertainment, and finance (Allen, 
2009). As the world increasingly relies on software-intensive systems, there will 
be an increased need for effective software project management in order to field 
successful systems. Ineffective software project management in these industries 


is among the main reasons for failures in software projects (Jones, 2004). 


Effective Software Project management is crucial to a software project's 
success. It was observed by DeMarco and Lister that for the overwhelming 
majority of bankrupt projects, there was not a single technological issue to 
explain the failure (DeMarco & Lister, 1999). Another study in the last decade 
asserted that a project was never seen to fail for technical reasons. It was always 
human failures that caused otherwise good projects to grind to a halt (Robertson 


& Robertson, 2005). Despite these observations most software engineering 
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research emphasizes technical matters above behavioral matters (Glass, 2002). 


People and project management are the Achilles’ heel of software projects. 


So are software project managers just poor at their jobs and solely to 
blame for project failures? This surely cannot be the case as many project 
managers are outstanding professionals. But software itself is incredibly complex 
and so is the management of its creation. A striking proportion of project 
difficulties stem from people failing to implement known best practices (The 
Royal Academy of Engineering and The British Computer Society, 2004). To 
become effective at software project management requires the project team to 
learn certain practices until they become habits. Good project managers will 
continually seek ways to improve their methods and learn from experience. But 
changes in how software is managed do not come quickly or easily. Any project 
management improvement process needs to be approached deliberately and 
purposefully. Project managers need tools to help them improve their software 
project management. A tool that measures and monitors the effectiveness of 
software project management can be used to identify strengths and weaknesses 
and guide improvement of the software project management practices in place 
on the project. Improving technical processes alone cannot ensure a successful 


project outcome. 
B. STATEMENT OF THE PROBLEM 


Effective project management involves measurement. Project managers 
measure schedule, progress, expenditure, effort, productively. These 
measurements are made to take the pulse of the project, in order to improve the 
project's health, if need be. But since poor software project management can 
increase software costs more rapidly than any other factor, as Boehm declared, 
should not the project's management itself be measured and monitored? Garcia 
and Suarez stated that project management practices are considered the 
cornerstone of the software lifecycle (Garcia & Suarez, 2007). If the project 


management practices can be improved, then a project should increase its 
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chance of success. However, committing a project to a significant improvement 
effort requires a thorough understanding of where the project is and, perhaps 
more importantly, where the project needs to grow (Grant & Pennypacker, 2006). 
The problem with current project management appraisal methods is that they 
take a long time to make an assessment, they do not focus on people and they 
are targeted at the organizational level. For this reason, project managers are not 
completely equipped with the right software project management effectiveness 


measurement tools. 
ile Effort of Analyzing Project Management Practices 


Effective software project managers should appreciate a candid review of 
how a project is being conducted or was conducted. As humans, we learn from 
our mistakes and conducting a post-mortem analysis of a project is considered a 
best practice by many software professionals. This is one method for a project 
manager to analyze the effectiveness of the software project management on the 
project. However, it was found by Chemuturi and Cagely Jr. that the project post- 
mortem evaluation is often skipped (Chemuturi & Cagley Jr., 2010). The reasons 
for this could be that the time is considered better spent on other income- 
generating activities. A software project management’ effectiveness 
measurement tool could assist with the post-mortem activity and even reduce the 


time it takes to conduct the activity. 
2. Project Manager Performance 


In general terms. there are three types of categories of project managers: 
those that know the best practices and apply them, those that Know them and for 
whatever reason do not apply them, and finally those that do not know them. 
Surprisingly, there is an absence of collective professionalism in the industry, as 
well as inadequacies in the education and training of staff at all levels (The Royal 
Academy of Engineering & The British Computer Society, 2004). The software 


project managers’ network asserts that a big problem in software projects is an 
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ill-equipped project manager (Phillips, 2000). A software project management 
effectiveness measurement tool could help project management professionals 
identify practices at which their project is poor or even practices they do not use 
at all. Even experienced managers could benefit from this type of tool. Due to the 
pressure and fog of war of software projects, one can forget to apply best 


management practices. 
3. Maturity Models Lack a People Focus 


Currently, there are a number of Maturity Models in widespread use that 
can be used to appraise a projects processes and guide improvement efforts. 
While these models assist with improving some software project management 
processes, they ignore the people side of software development. The first 
maturity model that comes to most people’s minds in the software development 
industry is the Capability Maturity Model Integration (CMMI) brand. It seeks to 
make proven software principles part of the organization’s culture and is often 
used to rate organizations software development capabilities. To most people, 
there is little doubt that adopting the specific practices recommended by CMMI 
will improve an organization’s ability to manage software projects. However, 
technical processes alone cannot ensure a successful software project outcome. 
CMMI-DEV-v1.2 contains a process area that focuses on project management, 
but this process area is devoid of management practice related to people 
(Phillips, 2000). CMMI-DEV-v1.2 focuses on an _ organization’s technical 
processes and not its highly unpredictable and behavioral components—people. 
The project management practices in CMMI-DEV-v1.2 are only one compartment 
of the greater software project management framework. This concept is 
illustrated in Figure 1. Software project management is about people and not just 


technical processes. 


Software 
Project 


anagement 
st Practices 





Figure 1. Software Project Management 


C. BACKGROUND AND NEED 


There is evidence to suggest that deficient project management practices 
may be one of the principal causes of software project problems. As such, there 
has been a widespread investment in project management education and tools 
as organizations strive to become good a delivering projects successfully (Grant 
& Pennypacker, 2006). There has been avid interest in the creation of models 
that provide a collection of best practices that managers can compare to their 
organizations’ practices in order to guide improvement. The front-runners in this 
area are the Project Management Maturity Models (PMMM), but there is also 
promising research in more lightweight software project management 


measurement tools. 
iF Project Management Maturity Models 


Maturity Models have spread quickly across the globe in the last two 
decades. From the foundation established by CMMI, new models, dubbed 
PMMMs, have immerged to focus on the project management maturity of 
organizations. The majority of PMMMs work in a similar way to the CMMI 
models. PMMMs, however, are concerned with generic project management and 
do not focus specifically on software project management. Software project 


management is more different from traditional project management than most 


S) 


professional managers expect (Fairley, 2009). There are, in fact, very few 


software Project Management Maturity Models (SPMMM) in existence today. 


While an SPMMM provides a means to assess the level of the software 
project management effectiveness, it does have a few limitations. A maturity or 
capability level can only be obtained after an independent, outside group 
examines the organization against specific criteria. To make an appraisal of an 
organization usually requires preparation, on-site activities and, finally, reporting. 
This takes a considerable amount of time. Additionally, maturity models claim to 
be able to target specific projects but are really focused at the organizational 
level and although maturity models have exploded onto the market there are 
many organizations that are still not using them (Garcia & Suarez, 2007). Due to 
these limitations, there is a need for more tailored, lightweight software project 
management effectiveness measurement tools. A lightweight appraisal tool can 
be used in a lot less time than a maturity model and can identify ineffective 


project management practices in place on a software project. 
2. Software Project Management Effectiveness Metric 


One such lightweight measurement tool was proposed by Demir in his 
dissertation entitled Measurement of Software Project Management 
Effectiveness (Demir, 2008). Demir proposed a Software Project Management 
Evaluation Model (SPMEM) that provided a standard quantitative measure of 
software project management effectiveness. The model accepted input data 
obtained from the application of a questionnaire to a software development 
project. It produced a standard quantitative measure, between zero and ten, by 
comparing the practices in place on the project to the best practices in the model. 
Demir measured sixteen software projects and produced a software project 
management effectiveness metric score for each. Pearson product moment 
correlation analysis was performed for the metric scores and a subjective rating 
of the projects success. It was found that there was a strong positive correlation 


with the project success rating and the software project management 
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effectiveness metric score. In addition, half of the variation in project success 
could be explained by the metric. Both of these findings indicate that the metric 


has a strong practical and theoretical foundation to build upon. 


The measurement takes significantly less time to perform than a maturity 
model appraisal and can be used to assist in the postmortem activity of a 
software project. The measurement can identify weak project management 
practices on a project and can guide future improvement efforts. It can guide 
managers by providing a quick assessment of how the project stands against 
software project management best practices contained in the model. When the 
tool is used to measure and monitor a project, it can act as a reminder not to let 
certain practices fall by the wayside. It can also provide objective proof of the 
project's deficiencies so as to prove to stakeholders what improvement efforts 


must be made and should be resourced. 


The Software Project Management Effectiveness Metric, while promising, 
is still in a developmental stage. The sample size of 16 projects used in Demirs 
study is not statistically significant. In addition, the previous sample included very 
few failed projects. Conducting further measurements using the tool will provide 


more insight into the applicability and limitations of the metric. 


D. PURPOSE OF THE STUDY 


il Purpose Statement 


The purpose of this study was to measure the software project 
management effectiveness of software projects using the SPMEM in order to 
increase the pre-existing sample size and reassess the correlation between the 
software project management effectiveness metric score and the subjective 


project success rating. 
The hypothesis to be tested is: 


The success of a software project positively correlates to its 
software project management effectiveness metric score. 


/ 


lf having a high software project management effectiveness metric score 
is associated with a high project success rating, it would indicate that improving a 


project's score would improve the project's chance of success. 
2. Need/Rationale for the Study 


The software project management effectiveness metric has the potential 
to assist project managers who are put in charge of software intensive system 
developments. The metric can assist with the post-mortem analysis of software 
projects, via identifying areas for improvements on subsequent projects. The 
metric can provide quantitative evidence to support improvement process 
decisions rather than just going off of a project managers gut feel. This tool can 
be used to measure and monitor projects so that project managers do not let 


best practices fall out of favor on the project. 
3. Description of the Study 


In order to provide an assessment of the correlation between the project 
success rating and the metric, data from recent software development projects 
was collected. The data was collected using the Software Project Management 
Evaluation Instrument (SPMEl). The SPMEI, which is a comprehensive 
questionnaire, was administered to software project managers, technical 
managers, software developers and team leaders. The research subjects also 
provided a subjective project success rating. The data collected using the SPMEl 
was used as input to the Software Project Management Evaluation Model 
(SPMEM). This model used the raw data from the subjects’ responses and 
produced the software project management effectiveness (PME) score for each 
project. These two measures were used to test the research hypothesis. In order 
to understand the measure of association between these two metrics, a 
parametric correlation analysis was conducted. The testing of the hypothesis was 
conducted by analyzing the Pearson product moment correlation coefficient 


(PMCC) between the two measures. 


4. 


Expected Goals and Outcomes of the Study 


The goal of this study was to build upon the sample of sixteen software 


projects in Demirs SPMEI research. With a larger sample size, a stronger 


argument can be made to use or not use the metric. Another goal was to gain 


further insight into the usefulness and applicability the metric. 


E. RESEARCH QUESTIONS 


lt was stated by Jones that effective project management is a determinant 


in the success of the software projects (Jones, 2004). The purpose of the metric 


is to monitor and improve the effectiveness of software project management. The 


following questions will be addressed in this study. 


1) 


Will improving a project's PME score increase the project's chance 


of success? 


(a) 


(b) 


What is the relationship between the PME score (measured) 
and the project success rating (measured)? 

What is the relationship between the PME score (measured) 
and the size of the project (measured)? 

What is the relationship between an institution's CMMI level 
(measured variable) and the PME metric (measured 


variable)? 


What are software development practitioner's perceptions towards 


the practicality and usefulness of the metric? 


(a) 


What are software development practitioner's perceptions 
towards the manageability, meaningfulness, actionability, 
ambiguity, reliability, accuracy, timeliness and predictability 
of the metric? 


Will software development practitioners use the metric? 


F SIGNIFICANCE TO THE FIELD 


From the literature review conducted in this study, it became evident that 
the software engineering field contains only limited scientific work that addresses 
theories of measuring software project management effectiveness. The results of 
this study have helped substantiate the applicability and usefulness of the SPME 
metric. The projects surveyed in this study also benefited by receiving metric 


scores that identified areas of weakness in their software project management. 
G. DEFINITIONS 


Project Success Rating: A subjective ranking, on a scale of zero to ten, 
made by a member of the project team on the successfulness of a project (zero 


being a complete failure and 10 being a complete success). 


Effectiveness: Efficiency is doing things right. Effectiveness is doing the 


right things. 


Conceptual Framework: A set of theories widely accepted enough to 


serve as the guiding principles within a particular discipline. 


Project: A group of coordinated work activities and tasks that utilizes 
resources to achieve specified objectives within a prescribed time frame (Fairley, 
2009). 


Software Project: A project concerned with developing software for a 
software intensive system. Software intensive systems include one or more 


digital devices and associated software. 


Software Project Management: The collection of work activities 
concerned with planning and estimating, measuring and controlling, coordinating 


and leading, and managing risk factors for a software project (Fairley, 2009). 
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Best Practices: Best practices are reusable activities or processes that 
continuously add value to the deliverables of the project. Best practices can also 
increase the likelihood of success of each and every project. But while all that 
sounds good, there exists a fundamental question of who defines what is or is 


not a best practice (Kerzner, 2004). 


Process: The steps taken to develop software; a recipe for software. A 
way of accomplishing one or more work activities and tasks; typically involves 


procedures and the use of software tools (Fairley, 2009). 


Product: The product is the project's final outcome. Products include 


software, documentation, and training and maintenance services (Phillips, 2000). 
H. LIMITATIONS 


Further development and modification to improve the SPMEI and SPMEM 
were considered outside of the scope of this research. The SPMEI was applied 
to only nine projects due to the difficulty of finding suitable participants willing to 
participate. The study was conducted on a sample of convenience. Having a 
small sample size reduces the studies external validity because of the limited 


generalizability to other settings and groups. 
I. ETHICAL CONSIDERATIONS 


As this study involved human subjects, the research required approval 
from the Naval Postgraduate School's Institutional Review Board (IRB) to ensure 
that the research was conducted in an ethical manner. Due to the nature of the 
research, the risk to participants was considered low. A breach of a subject's 
confidentiality may have resulted in some embarrassment. Informed consent was 


obtained from all participants and the consent form is contained in Appendix B. 
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ll. LITERATURE REVIEW 


Many research initiatives have emerged that focus on the improvement of 
software development processes and the technology used during software 
development. However, one area often underestimated but crucial for every 
software development effort is project management. (Mandl-Striegnitz & Litcher, 
1998). Software developers cannot rely solely on technological advances to 
achieve better outcomes in the development of software products. Software 
development houses need to make significant advances in the way they conduct 
project management in order to achieve better results. Applicable and viable 
theories on software project management need to be discussed and developed, 
and models and tools need to be tested and put into practice. Only then can 
software projects achieve better outcomes. One of the most important steps, for 
personnel practicing project management, is to look in the mirror and identify how 
their software project management practices can be improved. This research has 
identified several tools available in open literature that assess and measure the 
effectiveness, quality and maturity of software project management practices. 
Before these tools are discussed, a brief theory of software project management 


measurement is presented. 
A. METRICS IN SOFTWARE PROJECT 


Metrics serve only one purpose. We measure to manage (Brotby, 2009). 
In the management of software projects, it is widely accepted as best practice for 
managers to measure different components of their projects. For instance, 
progress is measured using Earned Value Management (EVM), while a product's 
performance is measured by using Key Performance Indicators (KPI) and 
software metrics. Quantitative measurements are essential in software 
engineering and there is a constant effort from academia and industry to improve 


and discover useful metrics. 
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A software metric is a measure of some property of a piece of software or 
its specifications (Singh, 2009). To give an example, here are some software 


metrics in widespread use: 


° Number of Source lines of code 

e Faults per lines of code 

e Number of lines of customer requirements 
° Function Points 

@ Cyclomatic complexity 

© Program load time 


The goal of research concerning software metrics is to obtain objective, 
reproducible and quantifiable measurements of software products. Metrics, 
measures, and monitoring processes exist only to provide decision support 
(Brotby, 2009). These measurements can then be used on software projects to 
assist with schedule and budget planning, cost estimation, and software 
performance optimization. The measurements can also be used to predict trouble 


ahead, such as the popular faults per lines of code metric. 


However, simply measuring the technical aspects of the software itself is 
only one part of a much larger and complex project. Effective project 
management is also a determinant in the success of the software projects 
(Jones, 2004). Measuring and monitoring the behavioral and management side 
of a software project should also be able to assist in providing decision support. If 
a project can measure and monitor its software project management capabilities, 
then the project can take active steps to improve these critical practices. 
Measurement of one’s software project management effectiveness enables the 
improvement of practices that are known to lead to a greater chance of project 


SUCCESS. 
B. SOFTWARE PROJECT MANAGEMENT EFFECTIVENESS 


It can be strongly argued that the effectiveness of software project 
management contributes significantly to the outcome of a software project. But 
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just what is software project management effectiveness? Effectiveness is defined 
in the Merriam-Webster dictionary as the power to produce a desired effect 
(Merriam-Webster Inc, 2011). Based on this, the following definition is offered: 
Software Project Management Effectiveness is the power of the software project 
management practices in place to accomplish the objectives of the software 


project. 


In management, effectiveness relates to getting the right things done 
(Drucker, 1993). If the right software project management practices are in place 
and the practices are implemented well, then the software project management is 
effective. An alternate definition is: Software Project Management Effectiveness 
is the degree to which the right project management practices are in place to 


produce the intended or expected result of the software project. 


C. THEORY OF SOFTWARE PROJECT MANAGEMENT EFFECTIVENESS 
MEASUREMENT 


The theory presented in this thesis proposes that it is possible to measure 


software project management effectiveness by determining: 


° if the right software project management practices were in place 
during a project 

© how well the software project management practices were 
implemented 


The right software project management practices are reusable activities or 
processes that continuously add value to the deliverables of the software project 
(Kerzner, 2004). By implementing these practices, a software project can 


increase the likelihood of Success. 


A generic conceptual approach for measuring software project 
management effectiveness in this way is presented in Figure 2. This approach 
requires the development of a software project management framework that 
describes best practices. A data collection instrument must then be developed to 


comprehensively sample a project relative to the previously developed 
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framework. Data is collected using the instrument and analyzed in a systematic 
way by the software project evaluation model to determine the score of the 
project's management effectiveness. The project can then take action to improve 


areas in which it is deficient. 
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Figure 2. Conceptual Approach to Software Project Effectiveness 
Measurement 


1. Development of a Software Project Management Framework 


For the measurement of software project management effectiveness to 
work, there needs to be a perfect standard of software project management to 
measure against. While all that sounds good, there exists a fundamental 


question of who defines what Is or is not a project management best practice 
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(Kerzner, 2004). There is no one correct way to manage a project, due to the 
extremely complex and diverse nature of software projects. What works well as 
best practice for one may not work equally well for another. For the measurement 
to have practicality, there needs to be a framework of effective software project 
management practices that, if implemented by a project, will increase the chance 
of success. There are various bodies of Knowledge on the theory and practice of 
software project management that can be used to develop such a framework. 
The development of a framework for software project management is the first 


step in creating an objective and repeatable metric. 
2. Development an Instrument and Evaluation Model 


secondly, a data collection instrument(s) must be developed to sample 
the software project management practices of a project in a representative and 
comprehensive manner. In this study, instrument validity is the extent to which 
the data collection instrument samples the effectiveness of the software project 
management in a representative and comprehensive manner. Data must be 
collected so that it can be analyzed to identify if the project is performing 


practices as suggested by the project management framework. 


There are a variety of data collection instruments and methods that can be 
used for examining a software project. These include questionnaires, interviews, 
and documentation reviews, to name a few. Each has its own advantages and 
disadvantages. Questionnaires were the most commonly used instruments 
observed in this literature review. This is mainly because questionnaires can be 
applied to many people and projects in a cost effective manner. Questionnaires 
are not as invasive as an interview and can provide quantitative data that can be 


analyzed promptly. 
3. Collecting and Analyzing Software Project Management Data 


After project data is collected by the instrument(s), it is analyzed to 


discover how well the software project management practices in place correlate 
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to the practices defined by the framework. The software project management 
evaluation model presented in Figure 3 is used to systematically analyze the data 
and produce a quantitative metric. The metric will give an indication on the 
effectiveness of the project management practices in place. Once it has been 
determined where the project is in reality compared to the suggested framework, 
a report can be generated to explain to the project where their project 


management deficiencies are. 
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Figure 3. Conceptual Black Box Diagram of Software Project Management 
Evaluation 





4. Making Improvements 


The project reviews the report and the metric produced in order to develop 
their own action plan that aims to implement new management practices or 
improve existing ones. By improving their practices they should improve their 
metric score. There are two ways that the metric could be used; and these are 
shown graphically in Figure 4. For a project that has a long duration, multiple 
measurements of the software project management effectiveness can be made 
at periodic intervals to ensure that improvements are being made. For a project 
of shorter duration, one measurement can be made at the end of the project as 
part of a post-mortem process so that improvements can be made by the project 


management and implemented on their next project. 
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Figure 4. Measurement Timings 


The theory of the measurement of software project management 
effectiveness is summarized in Table 1. A literature review was conducted on 
open sources to identify studies that are related to the concept of measuring the 
effectiveness of software project management as presented in this section. The 
literature review identified that very few studies have been published that 


concern this topic. Six such studies are summarized and discussed in the 


following section. 
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Table 1. Software Project Management Effectiveness Measurement 


Software Project Management Effectiveness Measurement 

What is being > The software project management effectiveness: 

measured? = The amount of “right” management practices that 

are in place? 
= How well the management practices in place are 
being implemented? 

Why is it measured? > The chance of software project success is greater 
when effective project management practices are 
done. 

What does itmean? » A high score means that the project has implemented 
the right practices and is performing them to a high 
degree in relation to the theoretical software project 
management framework. The project has a higher 
chance of success than a project with a lower score. 

> A low score means that the project has not 
implemented the right management practices in 
accordance with the theoretical software project 
management framework. The project has a lower 
chance of success than a project with a high score. 

Who are the > Software project management practitioners 

Recipients? 

What action is > The project team must implement changes to their 

required? project management practices in the areas where they 
are deficient in order to improve their chances of future 
project Success 





D. RELATED WORK 


1. Study: Software Project Management Maturity Assessment 
Model (2007) 


In their paper “Software Project Management Maturity Assessment Model 
to assess the level of Software Project Management Practices,” Fuazi and Ramli 
presented a model to assess software project management practices using their 
software Project Management Maturity Assessment (SPMMA) model (Ramli, 
2007). The purpose of the SPMMA was to help a company measure the strength 
and weaknesses of its software project management and develop action plans to 


make improvements. 
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a. Framework 


The SPMMA was developed using the concepts defined by the 
Capability Maturity Model Integration (CMMI) and _ Software Process 
Improvement and Capability Determination (SPICE) assessment models. The 
framework only focuses on the project planning, project monitoring and control 
and risk management process areas. The research was considered to be a pilot 
program and these three process areas are not deemed to be a completely 


comprehensive software project management framework. 
b. Data Collection Instruments 


There were two types of data collection instruments used in the 
model, a questionnaire and a set of interview questions. The questionnaire was 
used to gather data indirectly from practitioners. The questions were organized 
into groups of process areas drawn from the previously mentioned framework, 
such as project planning and risk assessment. The respondents could select 
from four possible answers for each question: yes, no, does not apply and don't 
know. An extract of the questions for the risk management section are presented 
in Table 2. 


Besides the questionnaires, interviews were used to directly obtain 
data on the software project management practices. The interview was used to 
give the assessor a better understanding of the project management practices. 
Related project management documentation was also reviewed to gain a more 
thorough understanding of the project. An extract of the interview questions is 


presented in Table 3. 
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Table 2. Questionnaire: Risk Management Section 


Questionnaire: Risk Management Section 

Are risks contingency activities planned? Yes No Does Dont 
Not Know 
Apply 

Does the project conduct meetings to identify Yes No Does Dont 

common causes of defects? Not Know 


Once identified, are common causes of risks Yes No 


prioritized and systematically eliminated? 


Does the project follow a_ written Yes No 
organizational policy for risks management 

activities? 

Do members of the software engineering Yes No 
group and other software-related groups 

receive required training to perform their risks 

orevention activities? 





Table 3. — ‘Interview Extract 


Interview Extract 


Please tell me about yourself and your experience as it relates to this project? 
Please describe your role and responsibilities on the project? 


How do you know what you are supposed to be working on? 
What training have you had for your job? 


Are you involved with any of the estimating and planning of the software project? 





C. Data Collection and Analysis 


The SPMMA was carried out on one mid-size Information 
Technology (IT) Company. Based on the questionnaire responses, interviews 
and discussions among the assessment team, a rating of fully implemented, 
largely implemented, partially implemented or not implemented was provided for 
each of the three process areas. Additionally, the assessment team produced a 
final report on the assessment findings and made _ improvement 


recommendations. 
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d. Results and Summary 


The pilot program received the following ratings for the three project 


management areas: 


6 Project planning — largely implemented 
@ Project monitoring and control — largely implemented 
@ Risk management — partially implemented 


A recommendation was made to the project to establish proper risk 
identification and contingency list. Fuazi and Ramli concluded that the SPMMA 
could be used as a tool to measure the level of maturity of the software project 
management practices in an organization (Ramli, 2007). While the SPMMA 
presents a method to measure the strength and weaknesses of an organization’s 
software project management there are a few concerns. First, only one project 
consisting of 40 personnel was assessed. Additionally, the tool only assesses 
project management in three areas and gives each area one of four possible 
ratings. The three areas in this framework are not considered comprehensive 
and the ratings do not provide much granularity. A summary of the model is 


provided in Table 4. 
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Table 4. Summary of SPMMA 


Summary of SPMMA 
What is being > The maturity of the organizations software project 
measured? management practices 
Why is it measured? > To help the organization measure the strength and 
weaknesses of its software project management 
and develop action plans to make improvements 
What does it mean? > Fully implemented - affirmation exists to confirm 
the implementation of the project management 
practice area and no weaknesses are noted 
> Largely implemented — affirmation exists to confirm 
the implementation of the project management 
practice area and one or more weaknesses are 
noted 
> Partially implemented — affirmations suggest that 
some aspects of the project management practice 
are implemented and one or more weaknesses are 
noted 
> Not implemented — no other evidence supports the 
conclusion that the project practice is implemented 
Who are the > Project management 
Recipients? 
What action is required? >» The assessed project has to prepare an action plan 
that specifies how, when and by whom each 
recommendation is to be implemented 





2. Study: What Project Management Practices Lead to Success 
(2005) 


Although not a measurement model, Verner and Evanco conducted 
relevant research using a questionnaire, in an attempt to determine the factors 
that lead to successful projects. They claimed that quantitative survey based 
research regarding software development’s early, non-technical aspects is 


lacking (Verner & Evanco, 2005). 
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a. Framework 


To develop their software project management framework, Verner 
and Evanco conducted wide ranging, structured discussions with 21 senior 
software practitioners to document views regarding the software project 


management practices they considered important. 
b. Data Collection Instruments 


A questionnaire was developed on the basis of these discussions. 
The questionnaire was organized into seven project management areas 
composed of numerous questions. An extract of the questionnaire is provided in 


Table 5. Respondents were also asked if they considered the project successful. 


Table 5. Verner and Evanco’s Questionnaire 


\V/=Ya a=) a= 1 ale A'e-laleye Meme 6(-s-) tle) alarclin= 
Did the project have a project manager? Yes No 


Was the PM above average? 
Was the PM experienced in the applications area? 


Did the PM communicate well with the staff? 
Were requirements gathered using a specific method? 


Were requirements complete and accurate at the project's Yes No 
start? 


Yes No 

icati Yes No 

| Yes No 
ifi Yes No 





C. Data Collection and Analysis 


In total, 122 in-house software development projects were analyzed 
using the questionnaire. The sample was not random, but rather a convenience 
sample of practitioners that Verner and Evanco knew. The sample size was very 
large for software engineering research of this nature and was the largest sample 
size discovered in this literature review. The variables in the survey were 


analyzed for correlation with project success and failure. 
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d. Results and Summary 


The objectives of Verner and Evanco’s research differ from those of 
the research in this thesis. Instead of attempting to measure the effectiveness of 
the software project management practices in place on a project the empirical 
research attempted to identify project management failures that lead to success 
or failure. It was found that a clear vision of the final product, good requirements, 
active risk management and post-mortem reviews can all help increase the odds 
of success (Verner & Evanco, 2005). For all projects, changing the project 
manager was. significantly negatively correlated with project success. If 
requirements were initially incomplete, completing them during the project was 
positively associated with success. Because software developers were surveyed, 
the results were limited to their knowledge, attitudes and beliefs regarding the 
projects and Project Managers with which they were involved. The method 
followed in Verner and Evanco’s research is an excellent way of developing a 


solid software project management framework. 


3. Project Management Maturity: An Assessment of Project 
Management Capabilities Among and Between Selected 
Industries (2006) 


Committing an organization to a significant improvement effort requires a 
thorough understanding of where the organization is and, perhaps more 
importantly, where the organization needs to grow (Grant & Pennypacker, 2006). 
One way to address this need is via the use of project management maturity 
models. The emergence of the project management maturity model can 
generally be traced to the Capability Maturity Model developed by the Software 
Engineering Institute (SEI) at Carnegie Mellon (Skulmoski, 2001). Project 
management consulting firms have played a leadership role in the development 
of many models, largely because the models are designed to identify areas upon 
which improvement efforts should focus. There are currently over 30 models in 


existence (Grant & Pennypacker, 2006). 
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A typical model works by assessing an organization's project management 
maturity. Once the initial level of maturity and areas for improvement are 
identified, the model provides a roadmap, outlining the necessary steps to take 
toward project management maturity advancement. Grant and Pennypacker 
conducted research to determine the level of project management maturity based 


on 42 detailed components among a wide range of industries. 
a. Framework 


The research conducted used the Project Management Solutions 
Incorporated’s Project Management Maturity Model (PMMM*™). The model 
adopts a two-dimensional framework, as shown in Figure 5. The first dimension 
reflects the level of maturity and is based on the structure of the SEI capability 
maturity model. The second dimension depicts the key areas of project 
management addressed. This dimension adopts the structure of the PMI’s nine 
knowledge areas (Project Management Institute, 2000). Each of the nine 
knowledge areas were further broken down into key components that provide for 
a more rigorous and specific determination of the project management maturity. 


There were 42 components in total. 


2/ 


B>vaA Level 1 Level 2 Level 3 Level 4 Level 5 
\ Initial Structured | Organizational Managed Optimizing 
Solutions Process Process Standards Process Process 


Project Management And And 
Maturity Model Standards | Institutionalized 
Process 







Project 
Integration 
Management (5) 


Scope 
Management (6) 
Time 
Management (5) 
Cost 
Management (5) 


Quality 


SEI 
Maturity Levels 










PMI 





Management (4) Knowledge 
Project 
Human Areas 





Resource 
Management (4) 


Communications 
Management (4) 
Risk 
Management (5) 


Procurement 
Management (4) 





Each Knowledge Area is broken down into specific 
components. Specific Components are used to measure 
maturity and develop action plans. The number of 


components associated with each knowledge area is 
presented parenthetically following the title of the 
knowledge area. 





Figure 5. Project Management Maturity Model (From: Grant & Pennypacker, 
2006) 


b. Data Collection Instruments 


A survey was generated that included a specific question for each 
of the 42 components of project management maturity. To ensure the content 
validity of the survey instrument, the CBP Knowledge Board reviewed it during 
the survey development process (Grant & Pennypacker, 2006). An excerpt of the 
survey is provided in Table 6. One advantage of this behaviorally anchored 
response scale format is that it has been shown to reduce leniency bias, or the 
tendency of a respondent to be overly generous or severe in evaluating 


organizational performance (Grant & Pennypacker, 2006). 
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Table 6. Cost Management: Resource Planning 


Cost Management: Resource Planning 

Level 1) Project managers have developed their own way of identifying resources 
and quantities needed; functional support areas are sometimes overlooked; 
process is not documented and varies by project. 

Level 2) Complete resource listing is for all labor categories, equipment, and 
material; planning process is developed and documented to include the resource 
listing and methodologies for determining quantities; planning process is 
supported by management and is becoming accepted throughout the 
organization. 


Level 3) Planning process is fully implemented within the organization; project's 
resource requirements are uploaded into the project office's resource repository. 
Level 4) All processes are in place, documented, and being fully utilized; process 
is fully integrated with the project office and the human resources project 
management 


Level 5) An improvement process is in place to continuously improve resource 
planning to completely identify all requirements as early as possible in the right 
quantities; lessons learned are captured and used to improve resource planning 
efforts. 





c: Data Collection and Analysis 


A total of 126 organizations were surveyed using a web-based 
survey. Each of the 126 respondents was asked to rate the project management 
maturity of his or her organization with respect to 42 specific components of 
project management maturity. Nearly 67% of respondents indicated their 
organizations were operating at level 1— initial processes (13.7%) or at level 2— 
structured process and standards (53.2%). While a notable portion of 
respondents rated their organizations as having reached level 3—organizational 
standards and institutionalized process (19.4%), a mere 7.3% indicated their 
organizations were operating at level 4—managed process and only 6.5% 
assessed their organizations as having achieved level 5—optimizing process 
(Grant & Pennypacker, 2006). 
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d. Results and Summary 


While this study does not rigorously measure project management 
maturity in the participating organizations, it did serve its purpose of collecting the 
ratings of numerous organizations’ project management maturity. However, this 
research focuses on _ organizational project management maturity and 
effectiveness, which is related remotely to project management effectiveness. It 
is also concerned with the higher level concept of project management and not 


software project management. 


Maturity in project management is the development of systems and 
processes that are repetitive in nature and provide a high probability that each 
project will be a success (Kerzner, 2004). It was found that there were many 
Project Management Maturity Models available to organizations wishing to 
improve their project management. These models focus on generic project 
management and do not specifically address the unique attributes of software 


project management. 
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Table 7. Summary of PMMM 


Summary of PMMM 
What is being > Where the organization is concerning their project 
measured? management maturit 
Why is it measured? > A thorough understanding of where’ the 
organization is and, perhaps more importantly, 
where the organization needs to grow is essential 
in order to make improvements 
What does it mean? > Level 1 - Initial (chaotic, ad hoc, individual heroics) 
- the starting point for use of a new process 
> Level 2 - Managed - the process Is managed in 
accordance with agreed metrics 
> Level 3 - Defined - the process is 
defined/confirmed as a standard business process, 
and broken down to levels 0, 1 and 2 
> Level 4 - Quantitatively managed 
> Level 5 - Optimizing - process management 
includes deliberate process 
optimization/improvement 
Who are the > Project Managers and Executive Management 
Recipients ? 
What action is required? >» Organization takes steps toward project 
management maturity advancement and 
performance improvement 





4. Study: Quality Management Metric (1999) 


Osmundson et al. (2003) developed a method, called the Quality 
Management Metric (QMM), to measure the quality of software management. 
The QMM is a composite score obtained using a questionnaire administered to 
both the program manager and a sample of his or her peers. The QMM is 
intended to both characterize the quality of software management and be used to 
improve an individual's and an organization's software project management 


Capabilities. 
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a. Framework 


It was proposed that the following four areas collectively were a 
suitable framework for the basis of a measurement of the quality of the software 


management in a project: 


@ Requirements management 

° Estimation & planning management 
® People management 

e Risk Management 


These areas were validated informally by experienced software 


professionals through the focus groups and one-on-one interviews. 
b. Data Collection Instrument 


The QMM was built to be an objective, repeatable metric to 
determine the quality of the software management, measure improvement, and 
predict future success levels of projects. A two-part questionnaire was developed 


to quantitatively measure the state of the software management quality. 


Table 8. Education/Planning Management 


Estimation/Planning Management: choose the most applicable term of the 
two for each row 


At least one estimation method used in No estimates 


program 
Formal derivation of product metric for Ad hoc size estimation 

estimation of size 

Ad hoc process evaluation Formal derivation of at least one 


orocess metric 


Develop work breakdown structure Assign work as needs arise 


Estimates are developed to fulfill a data Use estimates to plan program 
call onl 


Use estimates to sell program only Estimates are useful to the project 
team for planning purposes 


Expert judgment for estimation Ad hoc estimates 
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The questions were designed to confine responses. Part one of the 
questionnaire contained pair choice questions where the respondent had to 
choose one of two statements that best describe the project. An extract from part 
one of the survey is provided in Table 8. Often, the pair choices were repeated 
with different wording to confirm earlier choices and measure the strength of any 
tendencies. Part two of the questionnaire asks for one of three responses: yes, 
no or not applicable. This format standardized the response for easier 


comparison. An extract of part two of the survey is provided in Table 9. 


Table 9. People Management Questionnaire 


eX) o) (cM \UE-Tarle(cvpatsalmelecsr-yalelalar-ligs 


es No 
S No 


PM is accessible via phone by each team member ve 


PM acts as facilitator to solving personnel conflicts Ye 
PM attempts to spotlight individuals in the program for Yes No 
positive exposure 


PM maintains regular communication with users Yes No 
Yes No 


PM must aporove all interactions with users 


C. Data Collection and Analysis 





The survey was administered to 13 projects in the United States 
Department of Defense Environment. The projects ranged in size from three 
software developers to twenty-five software developers. The time frame of the 


programs surveyed range from 1992 to 2000. 


Each choice in the questionnaire had a point value assigned to it 
based on the relative importance of the question. Point totals for part one and 
part two were then added together to determine the total points for each area of 
software project management. The total points of each section were multiplied by 
its relative importance coefficient to yield a weighted score. After weighted scores 
were determined for each of the four sections, they were summed together to 
yield the QMM score. 

ChEM m C92Riqit + QO7TRRM + ESR KM + LEGPA 
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d. Results and Summary 


Each respondent was also asked to rate the success of their project 
on a scale of zero to ten. The calculated metric from each of the projects was 
compared to the subjective project success rating. This yielded a positive 


correlation with the subjective assessments of the project success. 


The QMM was the earliest research identified by this literature 
review to deal with the measurement of software project management 
effectiveness. The research showed promise but was limited by the sample set 
only consisting of Department of Defense projects. Additionally the projects were 


all during the 1990s, and the metric has been further validated since then. 


Table 10. Summary of QMM 


Summary of QMM 

What is being > Quality of software management 

measured? 

Why is it measured? > Improve organizations estimation process by 
including management quality as a program 
attribute 
Provide feedback to software program managers 
as to their management effectiveness 


What does it mean? Highest possible score — 100% - High chance of 
program success 
Lowest possible score — 0% - Low chance of 
program success 

Who are the > Project Manager 

Recipients ? 


What action is required? > Improve software management area with the 
lowest score 





5. Study: Two Phase Questionnaire (2007) 


Another questionnaire based-model was developed by Garcia and Suarez 
in 2007. Their approach sought to obtain a baseline snapshot of project 
management practices in small-to-medium enterprises using a two-phase 


questionnaire to identify both performed and non-performed practices (Garcia & 
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Suarez, 2007). The goal was to identify those practices that are performed but 
not documented, that practices need more attention, and which are not 


implemented due to bad management or unawareness. 
a. Framework 


To obtain an accurate picture of the project management practices, 
Garcia and Suarez based their framework on the Capability Maturity Model 
Integration for Development (CMMI-DEV) (Software Engineering Institute, 2006). 
The following seven well-established project management areas were used in 


the construction of the framework: 


° Project Planning 
® Project Monitoring and Control 
° Requirements Management 
e Configuration Management 
@ Process and Product Quality Assurance 
® Supplier Agreement Management 
@ Measurement and Analysis 
b. Data Collection Instruments 


A questionnaire was developed using closed questions as the main 
instrument for collecting data on the proposed framework. It was argued that the 
application of a questionnaire to an organization’s project team can provide 
useful information related to the current state of the project management 
practices and indicate those that required immediate attention. The questionnaire 
was divided into two phases. This division is mainly due to the fact that the 
CMMI-DEV clearly differentiates between specific practices and generic 
practices. Another reason for the division into two phases is because each 
section is applied to a different domain of people. The specific practices phase 


refers to the series of steps that have to be followed to perform the project 
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management practices. The generic practices phase refers to the maturity and 


institutionalization of the project management practices (Garcia & Suarez, 2007). 
C. Data Collection and Analysis 


The respondent could choose from the range of possible answers 
provided in Figure 6. Giving a specific weight to each response was proposed to 
enable the easy analysis of the results of the evaluation and identify which 
practices were common within the whole organization and which ones were not 


performed at all. At the time of publication, no such evaluation was undertaken. 


Possible Perform 


Description 
Answer Level P 


The activity is documented and established in the 

Always 4 organization. Jt is always realized, between /3 and 
100% ofthe time, in organization sojfrware projects 

The activity is established in the organization but 
rarely documented. It is usually realized, between 30 

and #5 % of the time, in organization software 
projects 
The activity is weakly established in the organization. 
itis realized sometimes, between 25 and 50% of the 
time, in organization sofmware projects 





TT, i 2 
Usually 3 


bog 


Somenmer 





The activity is rarely performed in the organization. It 
| is rarely realized, between J and 25 70 of the time, in 
organization software projects 
The activity is not performed in the organization. No 
Never Hf) person or group performs the activity in the 
organization. 


Rarely if 
ever 





Don 1 eee ee ee “a a ieee irae igs 
Fos The person is not sure how to answer the question. 
Not Apply fhe question is not applicable fo the oreanizatian. 
This space is for elaborating or qualifying one’s 
Comments response to a question, and it is mandatory when one 


selects Don't know or Not Apply options. 





Figure 6. Possible Responses in Two Phase Questionnaire (From: Garcia & 
Suarez, 2007) 


d. Results and Summary 


Garcia and Suarez felt that a more accurate picture of the project 
management practices of an organization could be obtained by administering a 
questionnaire. The next step in their research was related to the validation of the 


questionnaire. It was declared that in the future their questionnaire would be 
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administered to 26 small-to-medium enterprises through a project funded by the 


spanish Ministry of Industry, Tourism, and Trade. 
6. Software Project Management Effectiveness Metric (2008) 


The latest research known to cover the measurement of software project 
management effectiveness was published by Demir (2008). The metric 
developed by Demir sought to provide a standard quantitative measure of 
software project management effectiveness from the start of a project to its 
delivery. The objective of the metric was to helo managers in software 
development organizations to evaluate, monitor and improve their project 


management effectiveness. 
a. Framework 


A software project management framework was developed by 
Demir, and was validated by surveying 16 software projects. The framework 
consisted of 15 areas, which included: communication, teamwork, leadership, 
organizational commitment, project manager, stakeholder involvement, staffing 
and hiring, requirements management, project planning and estimation, project 
monitoring and control, scope management, configuration management, quality 


engineering, risk assessment, and risk control. 
b. Data Collection Instruments 


The Software Project Management Evaluation Instrument (SPMEl), 
which was a comprehensive questionnaire, was used to gather project data. The 
data collection tool was used to gather project data related to fifteen project 


management areas of the framework. 
C. Data Collection and Analysis 


Twenty software projects were assessed using the SPMEI in order 


to investigate the applicability and limitations of the metric. A member of the 


3/ 


project organization who had a broad knowledge on all aspects of the project 
management was asked to complete the questionnaire. Then the data gathered 
by the instrument was fed into the software project management evaluation 
model (SPMEM). Reponses to the questions were assigned specific scores in a 
similar way to the QMM mentioned previously. SPMEM simply combines these 
scores in a systematic way to produce a score for each project management 
area and these scores are then used to compute a software project management 
effectiveness (PME) score based on a scale from 0 to 10. A score of O indicates 
the least effective project management, while a score of 10 indicates the most 
effective project management. Each respondent was also asked to provide a 


subjective success rating from 0 to 10 in the same way as the QMM. 
d. Results 


The research provided empirical evidence required for the 
validation of the metric. A Pearson product moment correlation analysis on the 
data gathered showed that there is a strong positive correlation with success 
ratings and the software project management effectiveness metric. The result of 
the analysis on the data indicated that half of the variation in software project 


success may be explained by the project management effectiveness metric. 
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Table 11. Summary of SPMEM 


Summary of SPMEM 

What is being > Software project management effectiveness 

measured? 

Why is it measured? > Software project success is dependent on effective 
software project management 

What does it mean? > Highest possible score — 10 - A high PME score 


indicates a high probability of project success 
> Lowest possible score — 0 — a low PME score 
indicates a low probability of project Success. 
Who are the > Software project managers 
Recipients? 
What action is required? » Management takes steps to improve their software 


project management practices 





E. SUMMARY OF MODELS 


From the six studies reviewed, it was revealed that there is limited 
research on the topic of the measurement of software project management 
effectiveness. All of the studies reviewed are summarized in Table 12. Out of the 
six studies, only three provided an actual methodology to measure software 
project management effectiveness or maturity. These three models were all in 


early developmental stages. 
1. Framework 


Each study established a framework for software project management, 
even if it was not specifically called a framework in the study. Three of the 
frameworks were based upon the Software Engineering Institute's Capability 
Maturity Models. The others were based upon research and validated through 


peer reviews. 


The different software project management frameworks varied in content 
and comprehensiveness. There were, however, some recurring themes. 


Requirements management was considered important in four of the frameworks; 
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planning and estimation in five; risk management was present in four and 
monitoring and control stood out in three. The framework for the SPMEM was 


found to be the most comprehensive framework. 
2. Data Collection Instruments 


A constant across all the studies was the use of a questionnaire to gather 
data on the project management practices. In each study, it was argued that the 
application of questionnaires consumed less time, effort and financial resources 
than other methods of data collection such as interviews and document reviews. 
Another common theme was that the questionnaires were written in such a way 
as to minimize open-ended, subjective essay type answers. Of all the studies 
reviewed, only one used interviews and document reviews and that was to 


complement the use of a questionnaire in the data gathering process. 
3. Measurement 


The SPMMA only provides four possible ratings for the maturity of the 
measured project management areas. The QMM provides much more 
granularity, with the highest possible score being 100%. The SPMEM also 


offered a high level of granularity, with an ordinal scale of 0 to 100. 
4. Time to Implement 


To make the measurement usable by practitioners in the field, data needs 
to be gathered quickly and easily. The QMM was the quickest metric to 
implement at approximately 45 minutes, followed by the SPMEI at approximately 
90 minutes. The SPMMA took much longer to get a result. This was due to the 
interviews, documentation reviews and meetings that were required to make an 
assessment. 

5. Sample Size 

The three models that actually involve the measurement of software 


project management maturity are in their early stages of development. The 
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SPMMA was tested on only one project. The QMM was used on 13 United 
States Department of Defense projects. The SPMEI was applied to 20 projects of 


varying sizes and industries. 


The concept of the PMMM was extended by the SPMMA. This model 
focused specifically on software project management. However where these 
types of models focus on assessing the organization, the theory of project 
management effectiveness is concerned with measuring the’ software 
management of a single project within an organization. A large organization may 
claim to have a project management maturity level of 4 when they have multiple 
business units with hundreds of projects. Does this mean that every business 


unit and every project operate at a level 4? This is possible but not likely. 


At the completion of the literature review the measurement methods were 
subjectively ranked in order of effectiveness and potential for future use. The 
results are shown in Table 12. Out of the studies surveyed, the SPMEM showed 
the most promise for the measurement of software project management 
effectiveness. The framework and questionnaire developed were the most 
comprehensive and extensive. The measurements made thus far by Demir have 
shown a strong positive correlation with project success. The time to implement 
the questionnaire is reasonable and it has a strong sample base to build upon. 


The SPMEM is reviewed in detail in the following chapter. 
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Table 12. Summary and Ranking of Studies in the Literature Review 


OT voll atsl Moyers 1(=meyi 


WKexe (1 framework Instruments 
measurement 


Communication 

Teamwork 

Leadership 

organizational commitment 
project manager 
stakeholder involvement 


staffing and hiring i : 
requirements management Questionnaire 


project planning and 116 questions 


estimation 

project monitoring and control 

scope management 

configuration management 

quality engineering 

risk assessment 

risk control 

Requirements management : : 
Estimation and planning Questionnaire 
People management 

Risk Management 


Project Planning Questionnaire 


Project Monitoring and Control 


Risk Management Interview 


Project Planning 

Project Monitoring and Control 

Requirements Management 

Configuration Management ionnair 
Two phase Process and product quality Questionnaire 


assurance 
Supplier agreement 
management 
Measurement and analysis 
Project Integration 
Management 

Scope Management 

Time Management 

Cost Management 


Quality Management Questionnaire 
Project Human resource 1 
highagement 42 questions 
Communications 

Management 

Risk Management 

Procurement Management 
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In house 


Project Management 

Requirements elicitation and ‘ : 
management Questionnaire 
Cost and effort estimation and 

scheduling 

Postmortem 
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ll. REVIEW OF THE SOFTWARE PROJECT MANAGEMENT 
EFFECTIVENESS METRIC 


Of the studies reviewed in the previous chapter, Demir’s software project 
management effectiveness metric demonstrated the most potential as a software 
project management measurement tool. This chapter provides a more detailed 
review of the metric. The development and validation of the software project 
management framework used for the metric will be covered and the data 
collection instrument and the software project management evaluation model will 
also be discussed. A summary of the results obtained by Demir’s research will 


conclude the chapter. 
A. 3PR SOFTWARE PROJECT MANAGEMENT FRAMEWORK 


In Demir study, a simple software project management framework was 
developed that collected a set of software project management practices to serve 
as guiding principles for the software project management discipline. The 
framework was developed by an extensive review of the ubiquitous project 
management models, bodies of knowledge, standards and guidelines in 
worldwide circulation. To substantiate the developmental framework a survey 
was conducted on 7/8 software practitioners from around the world. Demir's 
framework consists of four main software project management areas: people, 
process, product and risk. 


® People. People management lies at the core of software project 
management and inclusion in the framework was mandatory. 
Thomsett (1995) pointed out that most projects fail because of 
people and project management concerns rather than technical 
issues. 


6 Process. The CMMI focus is on improving the maturity of 
organizations by improving their processes (CMMI Product Team, 
2006). The process main area focuses on key software project 
management processes. 


e Product. The software product is considered the outcome of a 
software project, which may be a product, service or result. The 
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objective of a project is to create a product with which the 
stakeholders are satisfied. This area is concerned with project 
management practices that focus their attention on the product 
quality. 


e Risk: Risk management is an inherent aspect of any software 
project. Boehm (1991) indicated that in most software project 
disasters, the problems could have been avoided or reduced if the 
high-risk elements had been identified and resolved early in the 
process. 


The framework consists of areas that can be measured. Each main area is 
decomposed into sub areas of project management. The sub areas give a higher 
level of granularity and assist in more refined measurements. Measurements in 
the sub areas can help project managers improve specific practices that are 
lacking. The complete framework is displayed in Figure 7 and is called the 3PR 


framework. 


Process 





Product 








Figure 7. 3PR Software Project Management Framework 


1. People - Sub Project Management Areas 


The people main area includes seven sub areas of software project 
management. They are communication, teamwork, leadership, organizational 


commitment, project manager, stakeholder involvement and staffing and hiring. 


a. Communication 


A successful project requires constant and effective communication 
between project stakeholders. It is a prerequisite to getting the right things done 
in the right way. Sharing knowledge empowers project stakeholders. Among all 
the project management areas listed in the Project Management Book of 
Knowledge, communications has the largest impact on project results (Muller, 
2003). 


b. Teamwork 


Teamwork is the process through which a collection of individuals 
cooperates to achieve an expressed common goal (Rasing, 2011). As software is 
developed by teams, strong teamwork is essential to successfully completing a 


software project. 
c Leadership 


In a software development environment, leadership is how 
personnel in management positions exert social influence to enlist the aid and 
support of others in the accomplishment of project goals. The thing great leaders 


have in common is the ability to get the right things done. 
d. Organizational Commitment 


In the framework organizational commitment is the employee's 
psychological attachment to the organization and organizational goals (Brown, 
2003). 


e. Project Manager 


The project manager position is a key role in a software project's 
organizational structure. A project manager should be a competent manager and 


leader. 
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f. Stakeholder Involvement 


The stakeholder engagement sub area is concerned with the level 
of involvement of all the different stakeholders during the project development 


effort. 
g. Staffing and Hiring 


In this framework, staffing and hiring is the ability to source human 
resources and put them in the right project role. Hiring is the process of 
employing personnel from outside the organization, whereas staffing is the 


process of sourcing personnel from within the organization. 
2. Process—Sub Project Management Areas 


This sub area includes requirements management, project monitoring and 
control, project planning and estimation, and scope management. These areas 


are more closely aligned to the process areas in the CMMI-DEV model 1.3. 
a. Requirements Management 


This process involves the management of the software 
requirements and is not to be confused with the requirements development 
process. Requirements must be controlled and consistency of requirements must 


be maintained with plans and work products. 
b. Project Monitoring and Control 


Comparing progress to plans and applying corrective action as 
needed. Project monitoring is the process of keeping the project, project-related 
factors and project metrics under continuous observation. Project control is the 
process of ensuring that a project goes according to what was planned. 


Deviations from the plan should be controlled and kept to a minimum. 
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C. Project Planning and Estimation 


Project planning involves establishing and maintaining the plans 
that define the project work activities. Software project estimation includes 
establishing estimates of project cost, schedule and resources using various 


methods, techniques and tools. 
d. Scope Management 


This is the process of defining the scope of the project and keeping 
track of any changes to the scope. Scope management was found in the 
validation of the framework, explained later, to be the most challenging sub area 


of the software project management framework. 
3. Product—Sub Project Management Areas 


This main area includes only two sub areas: configuration management 


and quality engineering. 
a. Configuration Management 


software configuration management is the discipline that enables 
us to keep evolving software products under control, and thus contributes to 
satisfying quality constraints (Estublier, 2000). Even though configuration 
management is a process, it comes under this main area because it focuses on 


the products developed by a software project. 
b. Quality Engineering 


Quality engineering involves all activities put in place to ensure the 
development of a high quality product. In this framework, quality engineering is 
not quality assurance. Quality engineering includes all the procedures and 
processes used to ensure products or services are designed and produced to 


meet or exceed customer requirements. 
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4. Risk—Sub Project Management Areas 


This main area includes only two sub areas; risk assessment and risk 
control. 


a. Risk Assessment 


Identify potential problems. According to Boehm (1991), risk 


assessment involves risk identification, risk analysis and risk prioritization. 
b Risk Control 


Develop and implement strategies and techniques for mitigating 
them. In order to conduct risk control, an effective risk assessment process has 
to be in place. Risk control involves risk management planning, risk resolution, 


and risk monitoring. 


Due to the nature of project management, the sub areas are closely 
tied to each other. For example, an effective risk control can only be achieved as 
a result of effective risk assessment. Effective teamwork can be achieved via 
effective communication, an able project manager, effective leadership of various 


leaders in the project organization and commitment from stakeholders. 
5. Validation of the Software Project Management Framework 


In order to validate the framework, a survey was distributed to software 
development practitioners to garner opinions on the framework. This form of 


empirical evidence was required to substantiate the framework. 


A self-administered questionnaire, which contained thirteen questions, 
was developed by Demir. The purpose of the questionnaire was to identify the 
importance of the software project management main areas and sub areas. The 
survey was also used to identify challenging areas of software project 


management. 


The survey was conducted in 2007 and was delivered to approximately 


400 software development practitioners. The sample was random and 80 usable 
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responses were obtained. It was found that all of the sub areas of the software 
project management framework were deemed to be important by the sample 
population. On a seven-point Likert scale, the average importance ratings ranged 
from a minimum of four to a maximum of six. This indicated that all of the areas 
were felt to be important by the sample population. The people sub areas were 
rated the highest and the process sub areas were the second highest. The 
product and risk sub areas were rated lower than the others, but were 


indistinguishable from each other. 


Additionally, participants were asked to rate the importance of the four 
main areas so that the total score added to 100. The mean of the ratings were 
the following: 

e People: 33.00% 

e Process: 29.07% 

e Product: 20.40% 

e Risk: 17.53% 


These ratings were used to adjust the software project management 
framework, as shown in Figure 8. The results of the validation study guided the 
development of the software project management evaluation instrument and 
evaluation model. 
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Figure 8. Adjusted 3PR Framework 


o1 


B. SOFTWARE PROJECT MANAGEMENT EVALUATION INSTRUMENT 


The goal of the SPMEI is to gather data on what happened during a 
software project. The instrument is not used for research as such but is intended 
to be used as a project management tool on software projects. The SPMEI data 
collection instrument is a self-administered questionnaire, consisting of the fifteen 
sections corresponding to the fifteen sub project management areas of the 3PR 
framework. Each section is comprised of a series of questions. Each question 
inquires about the effectiveness of an activity or an entity related to software 
project management. The complete instrument contains over 330 questions and 


is provided in Appendix B. 
1. Software Project Management Evaluation Instrument Design 


In every project, there are a set of software project management practices 


that: 
e should have been performed and were 
@ should have been performed and were not 
e should not have been performed but were 


The SPMEI investigates the project and collects data on all three of these 
scenarios. The instrument collects data on: 
® The existence of software management practices 


@ The rigor or quality of the practice 


Table 13 presents the different sections in the instrument and the number 
of questions in each section. In a questionnaire, questions can be classified into 
open and closed questions. The complexity of analyzing data provided by open 
questions is higher than those in closed questions (Yamanishi & Li, 2002). 
Closed questions provide less information but the results can be more easily 


interpreted in a measurement model. The questions in SPMEI are 
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closed for this reason. Closed questions also reduce the time required to 
complete the survey. No one wants to use a metric that takes an annoyingly long 


time to produce. 


The instrument covers the activities between the project conception and 
the delivery. Conception is the point where the project was established and 
funded. Delivery is the point in time where the final product Is delivered to the 


customer. 


Table 13. SPMEI Question Break Down 


Project Management Area Number of Questions 
Communication 23 
Teamwork 30 
Leadership 17 
Organizational Commitment 26 
Project Manager 27 
Stakeholder Involvement (Market or Contract) 12 or 16 
Staffing and Hiring 29 
Requirements Management 2/ 
Project Monitoring and Control 19 
Project Planning and Estimation 

scope Management 16 
Configuration Management 13 
Quality Engineering 20 

Risk Control 17 

Risk Assessment (With Subcontracting or Without 20 or 19 
Subcontracting) 

Total 330-335 





2. Application of the Instrument 


a. Who Can Use the Instrument? 


The metric is likely to be used by managers and organizations that 
are committed to achieving better results from their projects. These types of 
managers and organizations value candid assessments of their current practices 


and continuously seek to make improvements. 
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The instrument can only be used by a project member who has 
extensive knowledge and understanding of all the aspects of the project 
management practices. Generally, this type of person will fill the following roles in 


a software development project: 


6 Project manager 

e Team leader 

e Experienced developer 
e software architect 


b. What Projects Can Be Measured with the SPMEI? 


The instrument is only applicable to software intensive development 
projects. The instrument is not restricted to either public or private sector 
projects. The instrument is not applicable to corrective, perfective and adaptive 
maintenance efforts. Managing these sets of activities is different than managing 
development activities. The framework and instrument were not designed for 


software maintenance projects. 
C. Temporal Boundaries 


The instrument must only be applied to projects conducted after 
1980 (Demir, 2008). Until the nature of software projects changes dramatically, 
the instrument may continue to be used and improved. Demir speculated that the 
metric will be applicable for at least the next 15 years. The use of the metric is 


applicable to projects conducted from approximately 1980 to 2025. 
d. When Can the SPMEI Be Applied? 


The project must be established for a certain period before the 
instrument can be applied. The earliest that the measurement can be made is 


when the project has completed the initial requirements phase, or in other terms, 
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the inception or conceptual phase. By the time the project has reached this point, 
many of the essential project management related activities are already in place. 


To be specific, the following must be in place: 


@ the project manager has been chosen 

@ the project organization is identified 

e stakeholders have been identified 

® most planning and estimation activities have been carried out 

e the project scope has been established 

@ configuration management systems, project databases and other 
automated systems are in place 

° quality policy is in place 

@ project monitoring and control procedures should be in place 

° project communication procedures should be in place 

° An initial risk assessment has been undertaken 


Table 14. SPMEI Summary 


Name of the Instrument Software project management _— evaluation 


instrument 
Acronym SPMEI 


Main Use of Instrument Obtain data on what happened during the project 
development 
Type of Instrument Self-administered Questionnaire 
Participants > Project team members who have extensive 
knowledge of all aspects of the project. 
= Executive managers overseeing projects 
=» Project managers 
= Project technical managers 
= Team leaders 
Applicability software-intensive development projects 
Applicable to any project organization size 
Applicable with any software development life- 
cycle model 
Applicable to project after some requirements 
development activities are conducted 
Project start to project delivery (Project start is the 
time when the business decision is made) 
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Number of Sections 15 
Number of Questions 330-335 
Type of Questions > Multiple choice 


> Statements with a psychometric scale (5-point 
Likert item based on agreement to a statement) 
> All questions are closed form 


Time to complete Average of 90 minutes 





C. SOFTWARE PROJECT MANAGEMENT EVALUATION MODEL 


The SPMEM and the SPMEI were developed simultaneously. The 
software project management areas in the previously developed framework 
correspond to the variables in the SPMEM (Equation 1). The associated 
weighting of each variable was determined by the results of the framework 
validation survey. The variables in Equation 1 are calculated based on the data 
gathered from the SPMEI. For each of the variables (namely the software project 
main areas) there is an associated model to determine the value. Equations 2, 3, 


4 and 5 are used to calculate the main area scores. 
1. High-Level Evaluation Model 


The high-level evaluation model for the metric is as follows: 


(1) PME Score = 0.33People$ -+ 0.2907 ProcessS -- 0.204ProductS -|- Q17S32iskS 


where: 


PME Score: Software Project Management Effectiveness Score, 
PeopleS: People Area Score 

ProcessS: Process Area Score 

ProductS: Product Area Score 

RiskS: Risk Area Score 


2. Software Project Management Sub Area Evaluation Models 


The people main area score (PeopleS) is calculated as follows: 


(Op Pp bb OC > Pit > ST > F) 


(2) People Area Score = - 
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where: 


C: Communication Area Score 

T: Teamwork Area Score 

L: Leadership Area Score 

OC: Organizational Commitment Area Score 
PM: Project Management Area Score 

SI: Stakeholder Involvement Area Score 

S: Staffing and Hiring Area Score 


The process main area score (ProcessS) is calculated as follows: 


_ (RM + PMC + PRE + SM) 





where: 


RM: Requirements Management Area Score 
PMC: Project Monitoring and Control Area Score 
PPE: Project Planning and Estimation Area Score 
SM: Scope Management Area Score 


The product main area score (ProductS) is calculated as follows: 


(2) Product AreaScore 2 : OE) 


where: 
CM: Configuration Management Score 
QE: Quality Engineering Score 
The risk main area score (RiskS) is calculated as follows: 


(RA + RC} 


(5) Risk Area Score = 


where: 


RA: Risk Assessment Area Score 
RC: Risk Control Area Score 


3. Software Project Management Sub Area Evaluation Models 


The main area scores are derived from the sub area scores. The sub area 
scores are derived from participant's response to the questionnaire. For each 


response to a question in the SPMEIl, there is an associated score. The 
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associated scores for each response are provided in Appendix C. Adding all the 
scores together in a sub area provides an initial score for that section (or sub 


area). 


It is possible the initial score for a sub area will be a negative number as 
demonstrated in Table 15. The initial score tor each section is made positive by 
adding a shifting factor. This shifted score is normalized to a scale of 0 to 10 by 
multiplying it with a scaling factor. Table 16 provides an example of the shifting 


factors and scaling factors as derived from the values in Table 15. 


Table 15. Example Scoring Ranges 


People To) of Lowest al fefatexsy Difference 
Questions vsToxe) c=) veToxe) c=) 


Communication 23 -38 66 104 
Teamwork 30 -54 13 12/7 





The steps for calculating the score for a project management area are 


listed as follows: 


1) Sum the scores for each response in the section together. This is 
the initial score for the sub project management area. 

2) Add the shifting factor to initial score. This becomes the shifted 
initial score for the sub project management area. 

3) Multiply the shifted initial score with the associated scaling factor to 
normalize the score to a scale of 0 to 10. This normalized score for 
the sub project management area can now be fed into sub area 


evaluation model. 


Table 16. Example Shifting and Scaling Factors 


ed 0) (=Xea Management Shifting factor Scaling factor 
STU] oA =3- | 


Communication 38 10/104 
Teamwork 54 10/127 
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The generic model to determine a project management area score is: 





(4) Profect Management Sub Area Score = Scaling Facto 


In Equation 6, n is the number of questions in the section. PMA; is the sum 
of the scores for each response in a section. For example, in the communication 
section of the SPMEI, there are twenty-three questions. Thus, n is 23 for this sub 
area model. For the communication area score in Equation 7, the scaling factor is 
10/104 and the shifting factor is 38. For the complete details of the SPMEM refer 
to Appendix D. 
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D. SUMMARY OF RESULTS OF INITIAL STUDY 


sixteen software projects were surveyed by Demir. The graph below 
shows a plot of project Success ratings and PME scores. The trend suggests that 
there is a relationship between the PME score and the project success rating. At 
first look, it would appear that the higher the PME score the higher the project 


success rating. 
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Figure 9. Project Success Ratings and PME Scores 
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In order to understand the association between the PME score and the 
project success rating of a project, a correlation analysis was conducted by 
Demir. The Pearson product-moment correlation coefficient was used to identify 
the linear relationship between the sets of calculated variables. The Pearson 
product-moment correlation coefficient between the project success ratings and 
the PME scores is 0.68. This indicates a strong positive correlation between the 
two variables. Demirs study suggests that when project management 
effectiveness is high, project success is more likely. It was demonstrated that it is 
possible to develop a metric to measure the effectiveness of software project 


management. 
ii External Validity 


The small sample size of Demirs study is an obvious limitation and 
reduces the external validity of the study. It is difficult to make generalizations 
about the use of the metric on other projects with a sample size of sixteen. 
Additionally, there is only one project that has a lower project success rating than 
five and the subjects were only from America and Europe. Increasing the size of 
the sample and the range of success ratings should prove insightful. The goal of 
the research in this thesis was to increase the sample size by measuring more 
projects in order to provide further insight to the applicability and limitations of the 


metric. 
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IV. METHODS 


A. INTRODUCTION 


The development of a tool that measures the software project 
management effectiveness could prove to be highly valuable to software project 
managers. The Software Project Management Effectiveness Metric is one of 
these tools that have shown promise. To discover how promising the metric is, 


the following research questions were addressed in this study: 


1) Will improving a project's PME score increase the project's chance 
of success? 

(a) What is the relationship between the PME _ score 
(measured) and the project success rating (measured)? 

(o) What is the relationship between the PME _ score 
(measured) and the size of the project (measured)? 

(c) What is the relationship between an institution's CMMI 
level (measured variable) and the PME metric (measured 
variable)? 

2) What are software development practitioner's perceptions towards 
the practicality and usefulness of the metric? 

(a)What are software development _ practitioner's 
perceptions towards the manageability, meaningfulness, 
actionability, ambiguity, reliability, accuracy, timeliness 
and predictability of the metric? 


(bo) Will software development practitioners use the metric? 


To answer these questions, the research was conducted in two phases. In 
phase one, participants used the SPMEI to measure a project they had worked 
on and the SPMEM was used to obtain the PME score for their project. Phase 


two was a chance for participants to provide their feedback on 
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the metric through the completion of a short questionnaire. The data obtained in 
this study was combined with Demir’s for analysis. A visual depiction of the 


research method is illustrated in Figure 10. 
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<—"no consent] 


[participant consent] 
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- SPMEI data 


Analyse data with SPMEM 
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Participant reads PME metric report 


Participant completes Metric 
Feedback Instrument 


Combine data sets 
Test Hypothesis 


: Feedback data 


Report Research Results 


O 
Figure 10. Research Method (Activity Diagram) 
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B. SAMPLE/PARTICIPANTS 


1. Sampling Plan 


Research subjects for this type of study are most likely to be found 
through personal networking, via friends and colleagues (Demir, 2008). Due to 
the length and content of the SPMEI, potential subjects were recruited from the 
researcher's professional network. This was the sole means of recruiting subjects 


for the study and, as such, was a sample of convenience. 
2. Description of Participants 


In this study, a combined data set was obtained by joining Demir's data 
(henceforth referred to as existing data set) and the new data set, obtained by 
the research in this thesis. Nine projects were surveyed to create the new data 
set and the details of these projects are published in Table 17. The sample 
contains very recent projects of varying durations. The software products 


developed ranged from weapon systems software to web applications. 


Table 17. New Data Set Sample 


Project Delivery Date Software Product Duration (months) 


AA 2008 Command and Control 24 


BB 2010 Web Application 44 


CC 2010 Weapon System 29 
DD 2010 Command and Control 28 


EE 2011 Information and Data Management 28 
FF 2010 Entertainment NA 
GG 2010 Web Application 12 


Weapon System 
Web Application 





The combined data set contains 25 projects. The duration of the projects 
in the sample can be seen in Figure 12. The average project duration was 20 
months. The combined data set contains projects mainly from the last six years. 


The time frame for the projects is displayed in Figure 13. 
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Figure 11 presents the combined sample in terms of the average number 
of people involved. The projects are divided into four sizes: small, medium, large 
and very large. More than half of the projects in the combined sample are small 
size projects. One quarter is medium size and the remaining larger projects make 


up the rest of the sample. 


Very Large 
Large (20-100 (100+ People} 
People) ANG 
9% 





Figure 11. Project Size in Terms of Average People Involved 
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Figure 13. Project Delivery 
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C. MEASUREMENT INSTRUMENTS 


1. Phase 1: Software Project Management Evaluation Instrument 


The SPMEI was used for the first phase of the study and is included in 
Appendix B of this thesis. The SPMEI is scored by using the tables in Appendix 
C. 


2. Phase 2: Metric Feedback Instrument 


The second phase of the study used the Metric Feedback Instrument that 
was specifically developed for this research. The objective of this instrument was 
to obtain the subject's opinion on the usefulness of the metric. The instrument 
was created by using the eight attributes of good metrics as published by Brotby 
(2009). The instrument subjectively measures the manageability, 
meaningfulness, actionability, ambiguity, reliability, accuracy, timeliness and 
predictability of the metric. A description of each attribute is printed in Table 18. 
Each one Is subjectively assessed on a scale of one to ten. The participants are 
also asked to provide their own comments on each attribute of the metric and are 
queried to see if they would use the PME metric on their next software project. A 


copy of the instrument is provided in Appendix E. 
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Table 18. Attributes of Good Metrics (From: Brotby, 2009) 


PANiig ele) (: mum Bl-s-yergle)itela) 

Meaningful A metric must be understandable and relevant to the recipient 
and provide a basis for decisions 

Actionable Useful metrics information makes it clear what response is 
needed, as a compass makes it clear whether to turn left or right 
or stay on course 

Ambiguity Information from metrics can have a number of meanings and 
may be misleading, of little use, or downright dangerous 


Reliability The ability to trust the “instrument” is conditioned on the 
reliability of the measurement 


Accuracy A reasonable and known degree of a metrics accuracy is 
essential. The compass showing north when we are going 
south can be fatal 

Timely Measures that warn of a disaster after it has happened are not 
useful 

Predictive Some metrics information will signal impending problems much 


as a drop in oil pressure is the harbinger of engine failure 





3. Validity and Reliability 


The effectiveness of metric is the extent to which it provides information 
that meets the previously defined criteria for the recipient. The instrument 
produces a quantitative score out of 80 on the effectiveness of the metric (not to 
be confused with the software project management effectiveness). The 
instrument will also provide qualitative responses on the attributes of the metric. 
The metric feedback instrument provided a reasonably good and consistent 


measure of the metric’s effectiveness. 


D. DATA COLLECTION PROCEDURES 


Th Phase 1: Software Project Management Evaluation Instrument 


Potential subjects were contacted directly through the previously 
mentioned networking approach and informed of the study. If they were 


interested in participating, they were emailed a link to the SPMEI, which was 
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hosted online by SurveyMonkey. The participant connections to the online 
SPMEI were protected by VeriSign certificate Version 3 with 128 bit encryption. 
This provided assurance that participant responses were communicated securely 


to and from the SurveyMonkey servers. 


Risk. Due to the nature of the data obtained from the questionnaire, the 
risk to the subjects was deemed to be very low. A breach of the subject's 


confidentiality may result in some embarrassment for the subject. 


Consent. It was the investigators responsibility to obtain informed 
consent from the subjects before they commenced the survey. A waiver from the 


requirement to document the informed consent was obtained from the IRB. 


Data. The subject's data was retrieved from the SurveyMonkey servers 
and stored on NPS servers in order to conduct the research analysis. The 
researchers will ensure that the subject's confidentiality is maintained. No 


information was made publicly accessible that could identify the participants. 
2. Phase 2: Metric Feedback Instrument 


After the data was collected from the SPMEI, a metric was produced using 
the SPMEM for each project. The participant was then provided with a report on 
their project's PME scores. An example of this report is provided in Appendix F. 
The report maintains the subject's confidentiality. The instrument was distributed 


and data was collected in the exact same way as phase 1. 


E. DATA ANALYSIS 


a Phase 1: Software Project Management Evaluation Instrument 


Before any analysis was conducted, the PME scores for each project in 
the new data set was calculated and subsequently combined with the PME 
scores from the existing data set. The subjects recorded a project success rating 
at the start of the questionnaire and then again at the end. The average project 


success rating was used for the correlation analysis. 
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In order to determine the relationship between the PME score and the 
project success rating, a correlation analysis was conducted. The Pearson 
product-moment correlation coefficient (PMCC) was used to identify the linear 
relationship between the two measured variables. This analysis also allowed the 
researcher to test the hypothesis that the PME score positively correlates to the 


project success rating. 


The calculated PMCC, or r for this sample, will always lie between -1 and 
1. The polarity of r indicates the direction of the linear relation. In a positive 
correlation, when one variable goes up the other variable goes up as well. In a 


negative correlation, when one variable goes up, the other variable goes down. 


The absolute value of r indicates the strength of the linear relationship. 
The higher the value of r, the stronger the linear relationship between the 
variables is. When the absolute value of r is 1, this indicates that there is a 
perfect correlation between the two variables. Perfect relationships are rarely 
observed in social studies. In social studies, as a rule of thumb, when the 
absolute value of r is greater than 0.5, then it may be assumed that there is 
strong correlation between the variables (Demir, 2008). When r is below 0.5, the 
linear relationship between the variables is weak. A summary of the data analysis 
is described in Table 19. 
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Table 19. 


Research Question 
What is the — relationship 
between the PME — score 


(measured) and the _ project 
success rating (measured)? 


What is the — relationship 
between the PME _- score 
(measured) and the size of the 
project (measured)? 

What is the — relationship 
between an institution's CMMI 
level (measured variable) and 


success? 


Analysis 

Calculated the PME score for 
the 25 projects. Calculated the 
PPMC between the PME 
score and Project Success 
Rating 

Calculated the PPMC between 
PME score and the size of the 
project 


Calculated the PPMC between 
the PPMC between the CMMI 
level and PME score 


Will improving a project's PME score increase the project's chance of 


Data Collected 

25 Project Success Ratings 
and SPMEI data on 25 
projects 


Obtained data on the size of 
the project in terms of people 
involved 


Obtained CMMI levels for 9 
projects 


the PME metric (measured 
variable)? 





2. Phase 2: Metric Feedback 


The opinion data collected was categorized in terms of research questions 
and emergent themes. A coding method was used to organize data into a limited 
number of themes and issues around the questions. Quotations were then 


selected that illuminated the themes and concepts. 


Quantitative data analysis was also performed on the subject scores of the 
metric attributes. The results of the survey were analyzed using descriptive 
Statistics. The range, mean and standard deviation were obtained for each of the 
attributes. This statistics were also obtained for the total score for all of the metric 


attributes. 
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V. RESULTS 


A. PHASE ONE RESULTS 


1. Project Success Rating Results 


Participants in phase one subjectively reported the success of their project 
on a scale of 0 to 10 (0 for a complete failure and 10 for a complete success). 
Figure 14 is a histogram of the rounded project success ratings recorded for the 
combined data set. The mean rating, for the 25 projects in the combined data 
set, was 7.2. The mode of the project success ratings was 7. The smallest 
success rating was 2.5 and the highest was 10. If a score of 5 or above is 
considered to be a success, then 88% of projects were rated as successful by 
the participants. Projects with scores of 0, 1 or 2 were not represented in the 
sample. There were no projects sampled that were cancelled. The external 
validity of the sample would be increased if the lower range of project Success 


scores was increased in the sample. 
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Figure 14. Project Success Rating Histogram 
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Figure 15 presents the mean performance statistics of the projects 
(combined data set). On average, the projects delivered 97% of the required 
functionality, were 31% behind schedule estimated and were 23% over budget. It 
should be noted that not all projects reported their budget. The complete project 
Statistics are contained in Table 20. In some cases, there were significant cost 
and schedule overruns; however, the projects were still rated as a success. The 
project success rating is based on the eye of the beholder. If cost and schedule 
were not considered a priority, but functionality was crucial to success, then a 


project can still be rated as successful. 
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Figure 15. Average Project Performance Statistics 


Table 20. General Project Statistics 


Delivered 
Ulaveuleyat-liiay 


Software Category Duration KSLOC 


Project 


Delivery 
Date 


100% 


> 


2006 Information and 10 
Management 
2006 Embedded system 6 95% 
2009 Embedded system 17 100 100% 
2006 Embedded system 4 30 100% 
1995 Supply Chain Management 24 70% 
2008 Customer Service 12 95% 
1983 Command and Control 10 16 70% 
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2008 Command and Control 24 715% 
2010 Web Application 44 2000 150% 
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2. Software Project Management Evaluation Model Results 


This section contains various tables showing the sub area scores, main 
area scores and PME scores for all of the projects in the combined data set. 
Descriptive statistics are also contained in the tables. Table 22 presents the 
People sub area scores calculated using the SPMEM. Table 23 shows the 
Process sub area scores and Table 24 displays the Product sub area scores. 
Lastly, Table 25 shows the Aisk sub area scores and Table 26 contains the PME 


scores and project success rating. 


The People, Process and Product scores all had similar mean scores with 
6.6, 6.2 and 6.6 respectively. On average, the Risk area score was measured as 
the lowest performer with a mean of 5.6. This indicated that the projects in the 
sample all needed to work on improving their risk management practices. The 
range of main area scores and the PME score are all close to each other. Two of 
the projects obtained a score of 10 in different sub areas, indicating that perfect 
scores are possible. The minimum main area score was the risk area, with 2.5 


and the maximum was the product area with a score of 9.7. 


The lowest PME score calculated was 3.1, while the highest was 8.8. The 
mean of the PME scores was 6.3. Figure 16 is a histogram of the rounded PME 
scores, which has a mode of 6. It is important to highlight that every project in the 
combined data set with a PME score of 6 or above was successful. In other 
words, every project with a PME score of 6 or greater had a project success 
rating of 5 or greater. Table 21 shows the average project success rating for 
three different brackets of PME scores. It shows a distinct positive increase in 


success ratings as you move up through the brackets. 
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Table 21. PME Score Brackets 


PME Average Project Success Rating 


6 
>=6 and <7 7.8 





Table 22. Results of People Area Scores 
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DD 7.2 0.6 7.1 6.2 7.0 6.8 6.6 6.6 
EE 44 4.2 0.1 4.4 0.6 4/7 4.6 4.7 


FF 8.5 8.3 7.9 8.4 8.0 6.3 8.0 7.9 
6.2 0.f 9.6 0./ 6.9 0.1 0.9 5.9 
6.6 7.4 6.9 7.1 i 6.3 6./ 6.9 
3.0 24 2.1 3.9 2.9 0.4 83.1 3.1 


9.2 9.6 91 10.0 9.6 8.9 9.8 9.4 
6.3 [2 7.1 6.5 7.1 D.f = «6.7 6.2 


Standard Deviation 1.46 1.44 1.56 1.385 1.46 1.44 1.33 1.23 
Variation 2.12 2.09 2.44 1.83 2.13 2.07 1.77 1.50 





76 


Table 23. Results of Process Area Scores 


a ke) (=701 RM PMC a SM PROCESS 


A 7.0 7.6 6.4 6.9 7.0 
B 7.1 0.0 6.8 7.0 6.6 


fz 6.2 0.6 6.1 6.3 
0.4 6.9 6./ 0.9 6.2 


: 


Standard Deviation 1.49 1.27 1.17 1.48 
Variation 2.22 1.61 1.38 2.18 
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Table 24. Results of Product Area Scores 


Project QE PRODUCT 


A 
B 
C 
D 
E 
FE 
G 
H 
| 
J 
K 
L 
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N 
O 
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Standard Deviation 2.26 
Variation 5.12 
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Table 25. Results of Risk Area Scores 


Project 


A 
B 
C 
D 
E 
FE 
G 
H 
| 
J 
K 
L 
M 
N 
O 
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Standard Deviation 1.43 1.28 
Variation 2.04 1.64 
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Table 26. Main Area Scores and PME scores 


Project PEOPLE PROCESS PRODUCT RISK PME Success Rating 
7.9 7.8 7.2 8.0 7.8 9.0 
7.1 6.9 7.5 5.6 6.9 
9.4 8.6 9.5 7.0 8.8 9.0 
7.0 6.1 5.8 5.9 6.3 8.0 
5.7 4.9 7.9 3.9 5.6 3.5 
EE 4.7 5.4 4.9 5.9 5.2 6.5 
HH 6.9 7.0 6.8 6.3 6.8 10.0 
Max 9.4 8.6 9.7 8.0 8.8 10.0 
Range 6.2 6.0 6.0 5.5 95.6 7.5 
Standard 1.23 1.19 1.61 1.24 1.09 2.11 
Deviation 


Variation 1.50 1.41 2.60 1.53 1.19 4.43 
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Figure 16. Rounded PME Scores Histogram 


3. PME Score and Project Success Rating Relationship 


Figure 1/7 shows a plot of the project success rating and the PME score 
(sorted by the lowest success rating to the highest). At a glance, it would seem 
that the higher the project success rating the higher the PME score. An 
interesting phenomenon appears to be present as well. When the project 
success rating is 6 or below, the PME score is greater than the success rating. 
When the project success rating is above 6, the scores invert and the PME score 
is less than the success rating. It is difficult to make assertions about this trend 


with the current sample size. 
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Figure 17. PME Score and Project Success Rating (lowest success to highest) 


a. Hypothesis Testing 


The results of the PMCC analysis are contained in Tables 27 and 
28. The project success rating was graphed against the PME score in Figure 18. 
A quick look at this plot shows the likely existence of linear relationship between 
the PME score and the project success rating. The correlation between these two 
variables was found to be 0.68, which confirms the hypothesis: The success of a 
software project positively correlates to its software project management 


effectiveness metric score. 
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Figure 18. PME Score vs. Project Success Rating 
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Table 27. PMCC Between Sub Area Scores 


SM CM QE RA_ RC 
0.86 0.82 0.71 . . . . . 0.7/2 -0.09 


0.60 -0.03 
0.58 -0.01 
a 0.45 0.74 0.59 0.77 0.86 0.71 0.06 0.60 0.66 0.68 


= 035 045 O37 O57 O57 -0.14 0.18 0.57 0.45 
: wos 0.53 0./6 064 0.12 0.61 0.53 0.33 


uae 0.63 0.72 058 065 0.71 0.39 
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Table 28. 


PEOPLE 
PROCESS 
PRODUCT 
RISK 
PME 


PEOPLE PROCESS 


PRODUCT RISK 


PME 


Success 


CMMI 


PMCC Results for Main Area Scores, PME Score, Success Rating, CMMI and Staff Size 


Staff Size 


0.93 0.7/7 0.21 0.65 0.78 0.58 0.37 0.17 
0.96 0.82 0.37 0.63 0.85 0.58 0.66 0.22 


0.85 
0.86 
0.92 
0.61 


0.88 
0.76 


0.62 
0.65 


0.66 
0.69 
0.82 
0.56 
0.73 
0.84 


0.87 
0.93 


0.72 
0.85 


0.84 


* 


0.21 
0.36 
0.36 
-0.01 
0.41 
0.75 


0.44 
0.42 


0.75 
0.43 


0.31 
0.58 


* 


0.57 
0.37 
0.73 
0.56 
0.48 
0.61 


0.70 
0.85 


0.51 
0.92 


0.67 
0.85 
0.39 
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0.70 
0./2 
0.85 
0.51 
0.76 
0.86 


0.87 
0.87 


0.78 
0.82 


0.86 
0.97 
0.68 
0.83 


* 


0.62 
0.33 
0.69 
0.43 
0.50 
0.55 


0.53 
0.62 


0.35 
0.64 


0.63 
0.67 
0.33 
0.67 
0.68 


0.38 
0.56 
0.47 
-0.14 
0.43 
0.67 


0.46 
0.54 


0.47 
0.47 


0.46 
0.60 
0.49 
0.51 
0.62 


0.03 





4. PME Score and Project Size Relationship 


The correlation, r, between the PME score and the average project staff size was 
0.29. This indicates that there is not a linear relationship between the two variables. 
This inference can also be obtained from observing the plot in Figure 19. The graph in 
Figure 19 excludes the project that contained an average of 300 project staff in order to 


focus on the more concentrated data cluster. 
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Figure 19. PME Score vs. Average Project Staff Size 


5: PME Score and CMMI Level Relationship 


The correlation, r, between the PME score and a project's CMMI level was 0.62. 
This indicated that there is a possible linear relationship between the two variables. This 
result is also visually represented in Figure 20. This sample size only contained nine 


projects, which makes it harder to draw solid conclusions about this relationship. 
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Figure 20. PME Score vs. Project CMMI Level 


6. Other Correlation Analysis Results 


The correlation between the main area scores and the PME scores were all 
strong. The correlation between the process area score and the PME score in particular 
needs to be highlighted, as it is incredibly strong (r=0.97). This means that it could be 
possible to predict the PME score based on the process area score alone. This does 
not indicate that only achieving a high process score alone will give a high PME score 


because people, product and risk all contribute to the score. 


The correlation between the product score and the project success rating was 
0.33. The other three main areas all had strong correlations with project success 
(r=~0.65). 


The configuration management score had a poor correlation with success at 0.21 
and quality engineering was similar at 0.35. Organizational commitment had one of the 
lowest correlations with success (r=0.33). The project manager score had the highest 


correlation with the project success rating (r=0.69). Risk assessment and project 


8/ 


monitoring and control also had high correlations with success (r=0.64 and 0.66 


respectively). Improving these scores would suggest an increased likelinood of Success. 
B. PHASE 2 RESULTS 


The participants in phase one were all provided with their respective PME 
scores. After reviewing their PME scores, eight of the original participants provided 
feedback, using the Metric Feedback Instrument. The quantitative results are displayed 
in Table 29 and the qualitative responses can be examined in Appendix G. The average 
effectiveness score of the metric was found to be 59 out of 80 (SD=11.9). The individual 


scores for each response are presented graphically in Figure 21. 
1. Manageable 


For manageability, the metric scored a mean of 7. But due to the large range, 6, 
it would appear that opinions were quite divided over the manageability of the metric. 
The lowest score was 4 and highest was 10. No comments were provided by the 


participants on the metrics manageability. 
2. Meaningful 


The metric scored high for its meaningfulness (M=7, SD=1.7). It could be said 
that opinions were quite consistent over the meaningfulness of the metric. Opinions 
were generally positive, as echoed by one participant, “The survey seemed to translate 
well into scores | could relate to.” Another said, “It clearly defines the areas of good 
performance and the areas of concern.” However, another subject quoted, “The metric 
is meaningless without other data to support it.” This was interpreted to mean that the 
score alone is not helpful but, with supporting data such as the average PME scores, 
average sub area scores and average project success ratings, the metric could have 


more meaning. 
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3. Actionable 


The metric was considered actionable by participants (M=7, SD=1.7). The low 
variability in this score also indicates a strong consensus. It was noted by a subject that 
the areas where improvement was required was clear; however, it was hard to prioritize 
which area to target first. The subject stated, “Realistically, | am not going to be able to 
address each of the low scoring areas simultaneously, so if | have to pick an area of 
improvement, | want to pick the one that is going to give me the best chance of 
improving my project Success and that may not be the one with the lowest score.” 
Another subject asserted that when an area is performing poorly, by a large gap, 
compared to others it provides clear insight for improvement initiatives but in other 
cases it will be less clear what action to take. The metric does not currently provide 
specific data on questions in the SPMEI but one respondent provided an excellent idea: 
“In order to begin self improvement it would be good to see a breakdown of key 
techniques in each (sub) area and how you scored on each. That way you could begin 


focusing of (specific) techniques you were lacking in.” 
4. Ambiguity 


For ambiguity, the metric scored an average of 7 (SD=1.8). It was reported, by 
one participant, that the scores did not tell if they had done well or not. On a positive 
note, the sub areas satisfied another respondent, who commented that they created 


clear boundaries and that the sub area descriptions were simple to understand. 
5. Reliability 


It was pointed out by a subject that the reliability of the metric is inherently related 
to the reliability of the source. In other words, the respondent must have a thorough 
knowledge of the project management practices in place for the metric result to be 
reliable. One of the assumptions of using the tool is that it should be used by a person 
who has extensive knowledge on all areas of the project. The reliability score had a 
mean of 7 with a range of 4. 
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6. Accuracy 


The metric was considered to be accurate by the subjects (M=7, SD=1.6). One 
respondent found the metric to be very accurate and said that it reflected the weak and 
strong areas he instinctively felt the project had. The accuracy scores were the most 


consistent across all of the responses, shortly followed by reliability. 
t. Timely 


As a timely metric, the PME score was rated similarly to reliability (M=7, S=1.8). 
It was pointed out by a subject that if the PME score was produced after the initial 
requirements phase, then it would help the project manager grasp what type of project 
management activities still need to be carried out. This confirmed an_ original 
assumption that the measurement activity should be conducted after the _ initial 


requirements phase of the project. 
8. Predictability 


The metric was considered to have weaker predictive attributes by the subjects 
(M=6, SD=2.1). One participant commented that some of the sub area scores could be 
used in a predictive way, such as the stakeholder involvement score; however, other 
Sub areas were considered less predictive (i.e., teamwork). Another participant stated 


that they would not use the instrument as a predictive tool. 


Five out of six participants said they would use the metric on the next project they 
worked on. Although not seen as a particularly predictive metric, the majority of 
respondents found the metric useful. It was generally seen to be helpful in identifying 
strength and weaknesses. The low performing sub-project management areas could be 
selected for improvement action. It was also generally agreed that the measurement 
could be used to monitor the evolution of the software project management practices 
over time. On the negative side, the questions in the SPMEI were considered open to 


interpretation in certain areas. 
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Figure 21. Metric Feedback Scores for Each Participant 
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Table 29. Metric Feedback Quantitative Results 
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VI. DISCUSSION AND CONCLUSION 


With the complexity of software projects increasing every year, project 
managers need new tools to tackle these new system developments. A tool that 
measures the effectiveness of software project management could be used to 
identify the management strengths and weaknesses and allow projects to make 
improvements to their practices in order to increase their likelihood of Success. 


One tool that does this is the Software Project Management Effectiveness Metric. 


The purpose of this study was to measure the software project 
management effectiveness of recent software projects, using the software project 
management effectiveness metric, and obtain the opinions of practicing software 


professionals on the applicability and usefulness of the metric. 


Nine software projects were measured using the software project 
management evaluation instrument and a PME metric report was produced for 
each. A correlation analysis was conducted on the measured variables, PME 
score and Project Success Rating, combined with those from previous research. 
Six of the projects in the study reviewed their respective PME score and then 
completed a further survey that sought data on the practicality and applicability of 


the metric. 
A. DISCUSSION 


An important finding that needs to be highlighted is the relationship 
between the PME metric and the average staff size of a project. The correlation 
of this relationship was very low at 0.29. This shows that the metric does not 
favor projects of any particular size. This indicates that the PME can be used on 
any project size. However, a project manager should be most comfortable using 
the metric on projects with a staff size of at least four. This is because a more 
formal project management approach is typically used and required when project 


teams approach four or more. When the project staff size is below four it is 
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assumed that many project management practices in the framework would be 
unnecessary because the system development complexity would be less. For 
instance, a three-man web development effort may be a small business with no 
project manager, quality department or organizational hierarchy. It is 
recommended that the metric be used on projects with a staff size of four (or 


more) when a formal project management approach is required and in place. 


some noteworthy results were discovered about specific project 
management areas and practices. Firstly, the project manager sub area had the 
highest correlation (r= 0.69) with project success out of every single score. This 
corroborates well with Verner and Evanco’s pronouncement that an above- 
average project manager was positively associated with project success (Verner 
& Evanco, 2005). Secondly, the risk management main area was positively 
correlated with project success (r=0.67). In a similar way, Verner and Evanco 
surmised that managing risks throughout the project was significantly associated 
with project success. But Ironically, risk management was the least practiced 
project management discipline (Verner & Evanco, 2005). This was also found to 
be the case in this research. The average risk management score was 5.6 
(approximately one point below the other main area scores). Projects found to be 


deficient in these areas should concentrate their improvements efforts here. 


The relationship between the PME score and the project success rating 
was identified as having a strong positive correlation (r=0.68). The correlation 
found in this study's combined data set was exactly the same as the correlation 
calculated in Demir’s study. It was not expected to be the exact same value but 
the r value found in this study was expected to be above 0.5. This study has 
independently verified the strong correlation between these two variables as 


reported by Demir. 


The SPMEI itself was generally seen by participants to have a noticeable 
portion of ambiguous questions. One subject reported, “The (SPMEl) questions 
need to be less open to interpretation” and another said, “Reduce scope to 


questions that could be answered objectively.” It was suggested that some 
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examples integrated into the questions would remove the ambiguity. A good 
example of this type of ambiguity is present in one of the risk control questions, 
when the subject is asked if the risks are managed as they occur. A risk is a 
future event that may or may not occur. If a risk occurs, it is a problem impacting 
on the projects objectives. This type of question can be confusing. The SPMEl 


should be reviewed for ambiguity. 


This is the first study where the metric scores were provided to the 
participants and they were asked for their feedback on the practicality of the 
metric. It was found that 75% of respondents would use the metric on the next 
project they worked on. More research needs to be completed in order for the 
tool to be used a predictive measure. With more data, the metric can be studied 


to identify its predictive attributes. 
B. LIMITATIONS 


The external validity of the study is a weakness due to the small sample 
size. It was difficult to find participants to complete the SPMEI surveys even if 
they indicated interest during initial communications. Out of all the people 
contacted through the networking approach, there was a 53% SPMEl response 
rate. However, the combined data set of 25 projects now represents the largest 
sample size for the software project management effectiveness measurement 


tools covered in the literature review. 


Due to vast size of the software industry, it is fair to assume that the 
sample is not a fair representation of the software project population around the 
globe. At the same time, it is not possible to identify what a representative 
sample would be, due to the lack of published data about the software 


development industry. 


The correlation analysis depends on the accuracy of the PME score and 
the project success ratings. The project success rating is a purely subjective 


score. Subjects were asked at the start of the SPMEI to provide a rating, and 
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again at the end. In 66% of responses, the rating given at the end of the survey 
differed from the rating given at the start of the survey. This is an indication of 
how subjective the rating is and obviously the correlation analysis is affected by 
the subjective nature of the success rating. If this research was to be conducted 
again, it would be beneficial to have multiple opinions on the success rating of 
the project and then the mean could be used for correlation analysis. Another 


way could be to provide more objective criteria for project success ratings. 


Many participants skipped the essay-type questions posed in the metric 
feedback instrument. Additionally, many of the essay-type answers were difficult 
to interpret. If the feedback instrument was to be used again, a post-survey 
interview should be conducted to ask questions that respondents skipped and to 


clarify their answers. 
C. RECOMMENDATIONS FOR FUTURE RESEARCH 


The SPMEI sample size could still benefit from substantial growth. While 
building numbers is important, it is more critical for future research using the 
metric to concentrate on unsuccessful projects and projects with medium to large 
Staff size. Sampling these types of projects will fill a visible gap in the current 


sample and provide new insights to the lower end of the success spectrum. 


The SPMEI was not changed at all for this study. As mentioned 
previously, the SPMEI suffers from a degree of ambiguity in its questions. The 
SPMEI would benefit from a revision of the questions to decrease ambiguity. 
Additionally, the SPMEM score weightings for individual questions could be 
revised based on a correlation analysis of responses and the success ratings. 
This research could be conducted in a similar way to lvan and Evanco's study 


described in Chapter Il. 


The subjectivity of the SPMEI has still not been quantitatively analyzed. 


This has reliability and accuracy implications for the SPME metric. To garner 
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information on the subjectivity of the SPMEI, a study should be conducted where 
at least two personnel complete the SPMEI and a comparison of the results is 


made. 


The manageability of the SPMEI and SPMEM was a concern. In order to 
make the metric more manageable, the SPMEI can be broken down into its sub 
areas or main areas and distributed to different personnel on the project. The 
results can be combined and a PME score can then be produced. To test the 
applicability of this approach, one measurement can be obtained from multiple 
participants and another measurement can be made using a separate single 
participant. The two PME scores can be compared for accuracy. Splitting the 
SPMEI up into sub areas for completion shares the burden of completing the 
Survey among the project team members. Such an approach may require some 
redesign of the SPMEI and SPMEM as it was originally intended to be completed 


by one person only. 
D. CONCLUSION 


The present study illuminated some salient findings within the area of 
software project management effectiveness measurement. First, all the projects 
that scored a software project management effectiveness metric score of 6 or 
greater in this study were rated as a success. Out of the 22 successful projects in 
the study, 72% had a PME score of 6 or above. It was verified that the PME 
score had a strong positive correlation with the project success rating. From 
these results, it can be concluded that effective project management is a 
determinant in the success of the software projects. If a project has a PME score 
of six or greater, then they are on the right path to improving their probability of 


project Success. 


second, it was revealed by a correlation analysis that the metric can be 

projects with a wide range of staff sizes. Although it is recommended that 

projects have at least four members before applying the measurement, it is still a 

great tool for other relatively small projects who do not wish to invest the time 
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and effort in getting a CMMI appraisal. The metric can be used as a much more 
lightweight tool to improve project management practices. On the other hand, it 


could also assist with preparing for a CMMI appraisal as well. 


Lastly, probably the most important conclusion is that the currently 
practicing software professionals who took part in this study were exceedingly 
interested in using the metric on their next project. Seventy-five percent of 
respondents said they wanted to use the metric. It can safely be assumed that 
this tool needs to be put into practice immediately and, based on the results, 
project managers should be aiming to achieve a PME score of at least six as 
soon as practical. The practitioner feedback has helped to further substantiate 


the accuracy and usefulness of the SPME metric. 


software project management is a relatively new discipline, having only 
emerged in the latter half of the last century. A new discipline requires new tools. 
Like any metric, the software project management effectiveness metric should 
not be the one and only metric used on a project. But project managers should at 
least consider putting it in their tool kit. A metric that measures the effectiveness 
of software project management can be used to evaluate, monitor and improve 
the project management practices. This metric can clearly be used to identify the 
strengths and weaknesses of current project management practices and produce 
meaningful quantitative results. The metric shows the most promise as a post- 
mortem tool. Post-mortem reviews are important for process improvement, but 
projects seldom perform them. As a result, they tend to repeat the same 
mistakes project after project. This metric could be the awakening that some 


software project managers need, and a gateway to more success. 
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APPENDIX A. GLOSSARY 


Term DT =x-Yergle)iteyal 

Communication lt is the exchange of ideas, opinions and 
information through written or spoken words, 
symbols or actions. 

Configuration A discipline applying technical and administrative 

Management direction and surveillance to (1) identify and 
document the functional and _ physical 
characteristics of a configuration item, (2) control 
changes to those characteristics, (8) record and 
report change processing and implementation 
status, and (4) verify compliance with specified 
requirements. 

Leadership The ability to lead, including inspiring others in a 
shared vision. Leaders have clear visions and 
they communicate these visions to_ their 
employees. They foster an environment within 
their companies that encourages risk taking, 
recognition and rewards, and empowerment 
allowing other leaders to emerge. 

Organizational Organizational commitment is the employee's 
Commitment psychological attachment to the organization and 
organizational goals. 

Process A sequence of steps performed for a given 
purpose; for example the software development 
DrOCcess. 

Project Monitoring & Project monitoring is the process of keeping the 

Control project and project related factors under 
observation. Project control is to ensure that 
project goes according to what is planned and 
deviations from the plan kept under control. 

Project Project planning is the process to quantify the 

Planning/Estimation amount of time and budget a project will cost. The 





purpose of project planning is creating a project 
olan that a project manager can use to track the 
progress of his team. Estimation includes creating 
estimates of project cost and schedule using 
various tools and techniques. 


Quality Engineering In engineering, quality control and quality 
engineering are involved in developing systems to 
ensure products or services are designed and 
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Requirements 
Management 


Risk Assessment 


Risk Control 


Scope Management 


Software Project 
Management 
Effectiveness Metric 


Staffing & Hiring 


Stakeholder Involvement 


Supplementary Activities 


produced to meet or exceed _ customer 
requirements. It involves all activities and 
commitment towards development of a_ high 
quality product to meet or increase the 
customer/user satisfaction. 

The management of all requirements received by 
or generated by the project, including both 
technical and nontechnical requirements as well 
as those requirements levied on the project by the 
organization. 

A process or a set of activities that involves 
measurement of risks to determine priorities and 
to enable identification of appropriate level of risk 
treatment. 

That part of risk management which involves the 
implementation of policies, standards, procedures 
and physical changes to eliminate or minimize 
adverse risks. 

scope management is the process of keeping 
track of scope changes and limiting the changes 
to the point that they are not disruptive to the 
success of the project. 

This metric is a measure of the project 
management effectiveness in a software project. 
lt captures the effectiveness of the project 
management from the start of the project to the 
ooint in time of the measurement. 

Staffing is the practice of finding, evaluating, and 
establishing a working relationship with future 
colleagues on a project and firing them when they 
are no longer needed. Staffing involves finding 
people, who may be hired or already working for 
the company (organization) or may be working for 
competing companies. 

Stakeholder involvement is the early and 
extensive engagement of stakeholders in the 
process of planning, decision making, and 
implementation of a project. 

Supplementary activities are activities conducted 
which are not directly related to the project 
outcome. However, these activities indirectly 
increase the success probability of the project. 
Such activities include use of _ project 
management, development, testing and other 
types of tools, training of the personnel, logistics, 
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Teamwork 


Technical Complexity 


increasing the’ satisfaction of the work 
environment etc. 

Teamwork is the concept of people working 
together towards a common goal set as a team. 
Technical complexity refers to the complexity of 
the design, product, project deliverables and 
technologies used in the development of the 
oroduct. 
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APPENDIX B. SPMEIl 


Consent to Participate in Research 


Introduction. You are invited to participate in a research study entitled “The 
Effectiveness of Software Project Management Practices” being conducted by the 
Naval Postgraduate School. 


Procedures. The goal of this study is to gather information on software project 
management practices. You will be asked to fill out a questionnaire which will take 
approximately 90 minutes depending on the participant. The questionnaire is only 
related to the research and serves no purpose other than this research endeavor. 


Voluntary Nature of the Study. Your participation in this study is strictly 
voluntary. If you choose to participate you can change your mind at any time and 
withdraw from the study. You will not be penalized in any way or lose any benefits 
to which you would otherwise be entitled if you choose not to participate in this 
study or to withdraw. 


Potential Risks and Discomforts. The potential risks of participating in this 
study are: 


a) A breach of confidentiality may result in embarrassment of the research 
subject. 


Anticipated Benefits. Anticipated benefits from this study are: 


a) To assist in the development of project management metrics and improve the 
software engineering body of knowledge to improve software project 
management; and 

b) To enable the development of a tool for you to monitor, evaluate and improve 
your projects. 


Compensation for Participation. No tangible compensation will be given. A 
copy of the research results will be available at the conclusion of the experiment. 


Confidentiality & Privacy Act. Any information that is obtained during this study 
will be kept confidential to the full extent permitted by law. All efforts, within reason, 
will be made to keep your personal information in your research record confidential 
but total confidentiality cannot be guaranteed. No information will be publicly 
accessible which could identify you as a participant. Research records will be 
stored and maintained in electronic form on NPS secure servers only accessible by 
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the researchers. Any hard copy material containing research findings, including a 
thesis, will not contain any personal information. 


Points of Contact. If you have any questions or comments about the research, or 
you experience an injury or have questions about any discomforts that you 
experience while taking part in this study please contact the Principal Investigator, 
Dr John Osmundson, 831-656-3775, josmundson@nps.edu. Questions about your 
rights as a research subject or any other concerns may be addressed to the Naval 
Postgraduate School IRB Chair, CAPT John Schmidt, USN, 831-656-3864, 
jkschmid@nps.edu. 


Statement of Consent. | have read the information provided above. | have been 
given the opportunity to ask questions and all the questions have been answered 
to my satisfaction. | have been provided a copy of this form for my records and | 
agree to participate in this study. | understand that by agreeing to participate in 
this research and signing this form, | do not waive any of my legal rights. 


Dear Fellow Colleagues, 


| sincerely appreciate you taking time to participate in this study. This study is 
conducted as part of my postgraduate thesis research at the Naval Postgraduate 
school. My colleagues and | are testing the applicability of a software project 
management self-evaluation instrument (put simply, a questionnaire). We would 
like you to apply the instrument on a software project you have worked on. Your 
participation will be completely anonymous. 


How we plan to use your responses 
The anticipated benefits of this study are: 


a) to assist in the development of project management metrics; and 

b) to identify practices which increase the chances of project success; and 

c) to assist in the development of a tool for managers to monitor, evaluate and 
improve their projects. 


The only requirements for your participation are the following 


a) you have worked on a software intensive development project in the past; or 

b) you are currently working on a software intensive development project that has 
completed the initial requirements/inception/conceptual phase; and 

c) you have a broad knowledge of the project management practices in place on your 
project. 
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What personal information will be collected: 

The questionnaire investigates what happened during a particular project 
development. This is NOT an evaluation of the project manager, the 
management team, or any other person. This instrument is not designed for that 
purpose. Any inference derived for such a purpose will definitely be incorrect and 
misleading. This is NOT an evaluation of the organization. It focuses on the 
project only. 


How your response will be handled 

This study will be conducted with discretion and the highest regard for your 
confidentiality. In the final published research results it will not be possible to 
trace the results back to a particular person, organization, or any entity. Your 
response will only be identified as an identification code on all data collection 
forms. 


Your identification code is: XXX 
Please find the questionnaire attached. If you have questions about the study or 
the research, please do not hesitate to contact me. 


Yours Sincerely, 


Christopher Cullen 

Flight Lieutenant 

Computer Science Department 
Naval Postgraduate School 
Monterey CA 93943 


Tel: 1-831-917-5255 
Fax: 1-831-333-9277 


Email: ccullen@nps.edu 
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DIRECTIONS FOR FILLING OUT THE QUESTIONNAIRE 


Y There are 16 sections in the questionnaire. It takes approximately 90 minutes to 
complete, depending on the participant. The questionnaire examines from the start of 
the project until it is delivered to the customer for the first time (or it 1s cancelled). 


Y Choose a project you have worked on and have extensive knowledge. The project you 
choose does not have to be a complete success — it may have had moderate success, 
poor success or could even have been cancelled. We are interested in analysing the 
entire spectrum of software projects. 


¥ You may respond to the questionnaire sections in any order you like and you do not 
have to complete the survey in one sitting. 


v¥ The questions are straightforward and designed to be simple and easy to understand. 
There are two main types of questions. In the first type, simply check one or more 
statements that apply to the project. 


Check the STATEMENT that applies to the project. (CHECK ONLY ONE) 

L]x 

bd y 

[| None 

Check the STATEMENT/SS that applies to the project. (CHECK ALL THAT APPLY) 
[x] x 

Lly 

1 Z 

[ ] None 


vIn the second type, simply check whether you agree or not on a particular statement. 


Completely Agree Neutral Disagree ; y Not 
SIl | STATEMENT Ty) 0 im 7 isagr “eee 





¥ When there are combined statements, consider them as one concept and respond as is, 
or take an average of the ratings for each of the statement. 


v The questionnaire is designed as a whole. Trying to infer results from just one or 
more sections will be misleading. 


VY Please respond to all questions. Thanks again for your participation! 
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GENERAL PROJECT-RELATED QUESTIONS (17 Questions — About 5 minutes) 
Directions: Please provide responses to the following questions to the best of your knowledge. 





ENTER THE CODE PROVIDED: 


What was the goal of the project? What kind of an application was developed? What were the deliverables? Please 
briefly state. 





PRO What was the title of the project (if there is one)? 


What was the projected/planned effort for the project? (in terms of 
man-month) 
PR4. | What was the actual effort for the project? (in terms of man-month) 
PR5. | What was the actual cost of the project? 
What was the projected/planned budget for the project? 
How long did the project take? 
*From start (or contract) date to delivery date 
What was the projected/planned schedule for the project? Months 
re. What was the start date of the project? (Month/Year) 
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How much of the functionality (or number of features) are delivered 
to the customer? (Between the initial baseline and the delivered 
product) 


How many people worked on the project? (Including the management, consultants/contractors, etc.) 


Requirements Phase 
Design Phase 
Implementation Phase 
Testing and Delivery Phase 


What kind of an organization developed the project? (government, commercial, open source community, 
government contract, etc.) Organization name? 


PRIG What was the CMMI level of the organization when the project was undertaken? 0- 5 
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How would you rate the overall success of the project? (0 being complete failure and 10 being the complete 
success.) 


0 1 2 3 4 9) 8 9 10 


6 7 
[| LJ UD U UD UU UU uu vo oo 


What is/was your role in the project? 





INDEX (You can click to jump to a section) 


Communication 
Teamwork 


Leadership 

Organizational Commitment 
Project Manager 

Requirement Management 
Stakeholder Involvement 
Project Monitoring and Control 
Project Planning and Estimation 
scope Management 

Risk Control 

Staffing and Hiring 
Configuration Management 
Risk Assessment 


Quality Engineering 
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COMMUNICATION Section (23 Questions — About 7-12 minutes) 


C1. Check the statement/s that applies to the project. (Check all that apply.) 
[_] Acommon glossary/terminology for the project is created. 
[-] Communication procedures adapts due to changing project environment. 
[_] Communication procedures are always followed as stated in the communication planning documentation (or similar document). 
L_] There is a project information distribution list (or a similar document) and it is maintained. 
= The project budget includes resources for communication and project information distribution efforts. 
None 


C2. Who are generally present in the project status meetings? (Check all that apply.) 
L_] Project manager 

L_] Project team leaders 

[_] Project team members 

[_] Customer/s and/or user representatives 

[_] Various stakeholders or stakeholder representatives 

[_] Executive management / Project sponsor 


C3. Which of the following/s is/are discussion items in project status meetings? (Check all that apply.) 
L_] Project schedule 

L_] Project budget 

L_] Project risks 

[_] Project staff problems 

L_] Important development events and/or accomplished project deliverables 

L_] Requirements 

[_] None 


C4. Which of the following/s does the project information distribution plan/list (or similar document) contain? (Check all that apply.) 
L_] Project information type/context (What will be communicated) 

[_] Recipients of various communication items (Stakeholders- who should receive the information) 

[_] Project related information distribution frequency 

|_| Timeframe of the relevant communication 

[_] Communication format and medium (How the communication will be conducted- reports, meetings, teleconferencing etc.) 
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L_] Responsible project staff for communication 
[_] Not available 















‘ arr ee ‘ Completel Agree Neutral Disagree | Completel Not 
The importance of communication is understood and established between stakeholders and Tae S . " ABplicabia 
project team members. There is commitment to good communication. [| [| aoe [| 

Completel Agree Neutral Disagree eee Not 
Stakeholders including project team members’ needs for various project data and Bee. : ‘ 9 ‘oun 
information are analyzed and identified. [| [| Disagree 
Completely Agree Neutral Disagree ae as = 
There have been communication problems due to various reasons. (ial = Z Disagree cai 
=a Agree Neutral Disagree Compe 7 
Communication is used as a means to resolve conflicts. al Disagree ne 
Completel zi ree am as ree Compe 
There are designated project team members and representatives of stakeholders responsible = z = - i 
for conducting communication. Disagree 
Completel = ree m_ an ree Compe = 
Cig | Communication procedures are documented and distributed to stakeholders and project = . : - i 
team members. Disagree 


oe i le a_ compe 7 
C11 | Communication and coordination for activities are planned in the project plan. ial cai 


Dial ree 


C a tel L 
The response and acknowledgement procedures are planned and documented in es = iste ines ara roa 
the communication procedures. Disagree 


L_] 
L_] 





L_] 
[J SU 


C al tel 
13 | The information needs of stakeholders and project team members are satisfied in a — a ies meee <A 
a timely manner through appropriate use of communications media. [] Cl CI CI Disagree 





: . . . | Di C letel N 
As a project manager or a project team member, | can easily communicate my va Repel) Seagene, [vegans Pe ge pe 


14 
C messages and | can be understood. peas 
A communications and project information/data management system with a = a ses Compe - 
C15 | essential capabilities are in place. (Such as databases, mail servers, or Disagree 
C pee _ 
Bye The project environment facilitates horizontal communication that is between es = — = ee ne 
peers. Disagree 





C al tel _ 
The project team operates in a virtual environment rather than on a face-to-face a = ie — eer ae 
a Oo }OoO}] oO} OY eee 
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Completely Agree Neutral Disagree | Completel Not 
C18 | The project status is visible to every stakeholder and project team member. fel q Z Z Biscregn | 
z . fr C letel 
cig | The project manager, management team, and team leaders are always accessible i a (A a ee 
to project team members in a timely manner. ] C] es] 7 Pleaulee 
: C letel 
Coo | When | report a project problem, | get timely acknowledgement that my message i ceca cc | ka a et 
has been received and understood. CI] [] a B Pisaglee 
i eae : r C letel 
C21 | Informal communications within the team and stakeholders are also an important va eee ||| Pree: Uf reneerree Ret ae ae 
part of project development environment. 7 Disagiee 
a= = = Disagree a ae Not 
The project environment facilitates free-format meetings for various purposes. ial 7 ae 
C] sree 
_— — = Disagree Compl = 
C23 | The project environment facilitates freedom in reporting of project problems. Ty] EH Disagree ia 
eee = 


TEAMWORK Section (30 Questions — About 10 minutes) 


T1. Which of the following/s are clearly documented in the project plan for each team member? (Check all that apply.) 
L_] Responsibility of the team member 

L_] Accountability of the team member 

[_] Authority of the project manager and team members 

L_] Reporting structure 

[_] Interfaces and/or communication channels 

[_] None 


é<3[8<s[J8<s 
L ls s/Lig é\Lis 2 | 


T2. How many project team members stayed with the project until the end according to the project staffing plan? (Check only one.) 
L_] All L_] Most [_] Some L_] None 


T3. Check the statement/s that applies to the project. (Check all that apply.) 
L_] Notable project accomplishments/milestones/deliverables are celebrated with social events or parties. 
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[_] There are problem-solving meetings with the attendance of relevant project team members and stakeholders. 

L_] Organizational culture encourages problem solving sessions with the attendance of project members. 

[_] When a project team member left the team or the member is removed, the rest of the team has understood the reasoning. 
[_] None 


T4. Which of the following activities are carried out throughout the project? (Check all that apply.) 
L_] Social events/parties 

L_] Team building training 

L_] Introduction meetings and parties 

L_] Reward and other types of ceremonies 

[_] Brainstorming and problem solving meetings and sessions 

L_] Meetings for self-assessment of team performance 

[_] None 









6 | @® | ® | @ | @ | NA 


Completely Agree Neutral Disagree Completely a 
The project is adequately staffed during the project development. il A oil “ree 

C =e ae ree oe as ree =e letel z= 
The organization structure and responsibility/task matrix are clearly documented ae ‘ : : eee bi 
and provided to project team members. 


a mi am .=e = a 
There are regular status meetings to self-assess the project team’s performance and ie wale —_ 


morale. 


ace = = Zs _— A 
There is an accepted shared vision for the project within team members. i a im 
= = = =a — = 
T9 Team members are involved in the project planning effort. Ty] Ail “rere 
C =o a ree ae ree ae letel z 
Team members are involved in decision-making process during project ae ° : : oe Aerts 
development. 










aun = = — as CI 
The project status is visible to team members. iil Z = Z pieagie’: .°|' MPRIcaple 
° ° ° ° ° Completely Agree Neutral Disagree Completely Not 
T12 In order to do the work effectively, all necessary project data and information is Abie Disagree Applicablé 
easily accessible to project members. [| [ ] [| [| 
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oe Completel Agree Neutral Disagree Completel Not 

Training opportunities are created and made available upon need or at the request a . : picsuiee. || UAppcaBIS 
of team members. [J [| [| [| [isi [ 
° ° Completel Agree Neutral Disagree Completel Not 

There are more experienced project team members than inexperienced team cee ? . picauee. | ppllcabie 
members. a |, Se) Jee ee ee ee 
° ° ° Completel Agree Neutral Disagree Completel Not 

The project environment facilitates teaming up inexperienced team members with pe : . picauee. || wapeicabis 
the experienced team members. [ [] [ [| iz lis 
P ° Completely Agree Neutral Disagree Completely Not 

Rewards for achievements are handed out justifiably and made the project team Aetce Disagres Applicable 
happy. i: L [ [ [et [| 
Completely Agree Neutral Disagree Completely Not 

There is trust and respect among team members i TJ q Z ae a 


project team. [I] CT] Ea] J [J L_] 
members. 
members. 
evaluated. 

‘ilo lo cm 
procedures are followed for the team members joining the team later. [] [| [| [| [a [| 
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Completely Agree Neutral Disagree Completely Not 
The project team is empowered with adequate resources to do their tasks in q = Z ei sn Ui 
b 


letel N 

Project priorities are always made clear via meetings, presentations and “ oon ic (aa hae st 
memos; priorities are not constantly changing. 

a = = _ —— = 
The project suffers from lack of communication and coordination. Ty] ar a 

a — _ = — = 
The project suffers from lack of leadership at various levels. al at a 

nae mi am nee “=e CI 
The project team consists of people who has worked together before. T] ai baie al 


LEADERSHIP Section (17 Questions — About 3-6 minutes) 


Completel Agree Neutral Disagree Completel Not 
The leaders at various levels promote competition rather than coordination within pe : : piace (1 deplete 
the project organization. 


Completely Agree Neutral Disagree Completely Not 
The leaders at various levels sets example for others. a = Z Z ie ie 

Completel Agree Neutral Disagree Completel Not 
After the creation of the shared vision for the project, the leaders at various levels pag : : Dieagice. ||: appliesbié 
maintain the vision. i [| [| [| [| [| 


Completely Agree Neutral Disagree Completely Not 
ia a nie 








The leaders at various levels are effective problem-solvers in technical and social 
issues. 


— i” i ae ao z 
The management protects the team from outside interference. ml cis cn 


at = ree = ne ree eo r 


The leaders at various levels clearly state their leadership styles upfront with ral — a 


reasons for the style. 


— a i ae neo x 
L7 The leaders at various levels assign correct tasks to correct people. mi ai or 

= m) nat ae ne z= 
L8 The leaders at various levels are respected by the team members. i = ie ne 

Completely Agree a: .=s ao s 
L9 The leaders at various levels easily delegates authority when necessary. Ty a ae a 
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SEE 
ay 
—) 


° ° Completel Agree Neutral Disagree Completel Not 
The leaders at various levels observe the morale of the staff and takes proactive pe : : picsuiee. || UAppcaBIS 
action to boost the morale. [| [| [| [| [in [| 
Completely Agree Neutral Disagree Completely Not 
L11 | The project team suffers from coordination problems. Ty] cai al 
=a ml am a = = 
The project team suffers from communication problems. Ty] ar ii 
a — a oz — A 
LB The leaders at various levels welcome communication of project problems at any - ee are 
time. 
. - ° ‘ =a Agree Neutral a == z= 
L14 The leaders at various levels clearly define what is expected from project team aed ol ie —_ nee 
members. L] 
A ‘ = * ° oS Agree Neutral — — = 
L15 The project team members freely share their desires, wishes, and concerns with = — mene 
their leaders. i} 
a Agree = ae _— = 
The leaders at various handle project politics well. Ty] = ae a 


* Provide response to either L17 or L18. 

L17. (Answer only if the project team mostly consists of inexperienced staff) Check the statement that applies to the project. (Check 
only one.) 

[_] The leaders at various levels have to make most decisions and direct the staff. 

[_] The leaders at various levels make most decisions with the consultation of team members and coach the staff. 

L_] The leaders at various levels and the team members make decisions together. 

L_] The leaders at various levels mostly oversee the decisions made by the staff and delegate the tasks. 





L18. (Answer only if the project team mostly consists of experienced staff) Check the statement that applies to the project. (Check only 
one.) ee 

L_] The leaders at various levels have to make most decisions and direct the staff. 

L_] The leaders at various levels make most decisions with the consultation of team members and coach the staff. 

[_] The leaders at various levels and the team members make decisions together. 

L_] The leaders at various levels mostly oversee the decisions made by the staff and delegate the tasks. 
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ORGANIZATIONAL COMMITMENT Section (27 Questions — About 7-12 minutes) 


Completely Agree Neutral Disagree Completely Not 
The executive management is committed to providing necessary financial support. al ail i 

Completel = ree a= a= ree == letel z= 
The executive management is committed to providing necessary flexibility on the = 4 : 5 nese sna 
project schedule. 

Completel = ree am az ree == letel z= 
The executive management is committed to providing necessary flexibility on the = ‘ : : nase ci 
project functionality and quality. 

— — um =e —— = 
The executive management and project organization is open to change/adaptation. ial a acl ah 

Completel = ree =_ a= ree == letel = 
There is encouragement for organizational and personal certifications such as = 5 2 : alee va 
CMMI, PMI, PMP, ISO etc. 


ane ml um aee = Zz 
There is commitment to quality by executive management, team members and i _ wan : 
other stakeholders. 


== im = aes == = 
Adequate resources are set aside for the success of the project. ial all ar 


ane mi um nee =e Zz 
There is support for bringing in expertise when needed (Such as technical, legal, = i — 
contracting etc.) 


acs im = ae == = 
There is support for quality subcontracting when needed. Ty ail i 
Completel = ree a a= ree == letel 2 
Oc10 | The executive management supports / empowers / enables the project manager to = 4 : 5 nae spe 
do his job. 
— im = =e == = 
OC11 | There is continuous and observable support from executive management. ial = = = 7 ar 


Completely Agree Neutral Disagree Completely Not 
OC12 | Leaders at various levels are committed to the success of the project. Pelee pisagies ||’ #spplcave 


Completely Agree Neutral Disagree Completely Not 
OC13 | Leaders at various levels are committed to their team members. ial Z = Z i cab ar 
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0c14 | The project manager and leaders at various levels are committed to providing apa ead Ore meee! Wee | ae 
continuous support in enabling the team members to do their work. [J [| [| [| [sy [| 
001s Completely Agree Neutral Disagree Completely Not 
OC15 | The project team members are committed to the accomplishment of the project. Ty] = = E cai al 
Ocig | Lhe project team members show their commitment to staying with the project until eae ad aa eee eee | neat 
the end. [] [| [] a Zz [J 
C17 } Te projet tem members put extra effort fr the sucess the project a o o o |e 
OC17 | The project team members put extra effort for the success of the project. Ty] = = E cai al 
Ocig | The project team members lack motivation due to various reasons including la eee [eee | neat oe 
external factors. [| [| [| [J nl [| 
OC19 The project manager and the team members don’t consider the project as a ie i oui a ee Applicable 
pleasant challenge. [| & [| [4 ic [| 
0C29 | The project manager and the team members consider the project as a valuable ce ied tela see [eee | neat oe 
learning experience. [| [| [J [| [inl [| 
Completely Agree Neutral Disagree Completely Not 
OC21 | There is a friendly-work environment. pee pisadiee:. | ( -Appllcahle 
LI LI] LI LI LI] LI 
0022 Completely Agree Neutral Disagree Completely Not 
OC22 | The project team members publicly and explicitly indicate their job satisfaction. al Z Z = eae a 
0C23 | There is commitment from various stakeholders including project team members, ees aad mre pees Wee | ea oe 
customer, marketing and sales department(if applicable) etc. [| a [| az [oa [] 
0C24 | Executive management, project manager and project team members are committed a ules le meee ee | aaa ae 
to establishing effective project management and control mechanisms. [J i [| [| fal [| 
OC25. Which of the following item/s does the executive management show commitment to providing support? (Check all that apply.) 
L_] Human resources 
L_] Training needs 


[_] Supplementary needs such as office space, tools, computer systems etc. 
[_] None 
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0C26. Check the statement/s that applies to the project. (Check all that apply.) 

[_] The executive management clearly defines the authority and responsibility of the project manager. 
L_] The executive management allows for realistic budget and schedule. 

L_] Training is made available to all team members. 

[_] There are some resignations in the project organization. 

[_] The project organization allows for career development. 

[_] None 


PROJECT MANAGER Section (27 Questions — About 5-9 minutes) 


PM1. How many project managers have changed during the project (Turnover)? (Check only one.) 


L_] None L]1 LJ2 (LJ3ormore 
PM2. How many years of experience does the project manager have? (Check only one.) 
[_] Less Than 5 [ | 5-10 [ ] 10-15 [ | 15-20 [_] More Than 20 


PM3. Check the statement/s that applies to the project. (Check all that apply.) 

[_] The project manager has certification related to project management such as PMP etc. 

L_] The project manager has worked on similar projects. 

L_] The project manager has worked as a project manager before. 

[_] The project manager has worked as a practitioner/developer before, therefore has technical background. 
L_] The project manager has worked on different types of projects. 

[_] None 


PM4. Which of the following/s the project manager has control over? (Check all that apply.) 
L_] Budget [_] Schedule —[_] Product Quality [_] Process Quality L_] Hiring and letting go [_] None 


(5) (4) (3) (2) (1) N/A 


° cys Se epee , Completel Agree Neutral Disagree Completel Not 
pms | Lhe project manager’s role, accountability, and responsibilities are clearly defined 7" : : nese ale 
and communicated to stakeholders including project team members. 
Completely Agree Neutral Disagree Completely Not 
PM6 | The project manager was given adequate authority and control over the project. ial q Z z “el “ree 
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The project manager has adequate project management education, training and 
experience. 


As a project manager, I have goals and a clear vision related to the project. /As a 
team member, I observe that the project manager has goals and a clear vision 
related to the project. 


As a project manager, I am able to maintain the continuity of the project vision. / 
As a team member, I observe that the project manager is able to maintain the 
continuity of the project vision. 


As a project manager, I am deeply committed to the project./As a team member, I 
observe the deep commitment in the project manager. 


As a project manager, I am communicative and always accessible to team./As a 
team member, I observe that the project manager is communicative and always 
accessible to the team. 


As a project manager, I motivate staff and other people well./As a team member, I 
observe that the project manager motivates the staff and other people well. 


As a project manager, I am a good planner and organizer./As a team member, I 
observe that the project manager is a good planner and organizer. 


As a project manager, I am an effective problem solver./As a team member, I 
observe that the project manager is an effective problem solver. 


As a project manager, I consult to and get advice from stakeholders and project 
team members. / I observe that the project manager consults to and gets advice 
from stakeholders and project team members. 


As a project manager, I delegate easily when necessary./As a team member, I 
observe that the project manager delegates easily when necessary. 


As a project manager, I use rewarding and punishment mechanisms effectively. /As 


a team member, I observe that the project manager uses rewarding and 
punishment mechanisms effectively. 
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Applicable 


Not 
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As a project manager, I am a people person./As a team member, I observe that the 
project manager is a people person. 


As a project manager, I am an effective team builder and player./As a team 
member, I observe that the project manager is an effective team builder and player. 


As a project manager, I support my team members in various aspects./As a team 
member, I observe that the project manager supports the team members in various 
aspects. 


As a project manager, I monitor every aspect of the project./As a team member, I 
observe that the project manager monitors every aspect of the project. 


As a project manager, I inform the stakeholders and my team members well./As a 
team member, I observe that the project manager informs the stakeholders and the 
team members well. 

As a project manager, I clarify when the stakeholders and the team members are 
confused about an aspect of the project./As a team member, I observe that the 
project manager clarifies when the stakeholders and the team members are 


As a project manager, I am able to see the project as a whole./As a team member, I 
observe that the project manager sees the project as a whole. 


As a project manager, I understand the domain of the project./As a team member, I 
observe that the project manager understands the domain of the project. 


As a project manager, I protect my team members so that their work don’t get 
disrupted./As a team member, I observe that the project manager protects us so 
that our work don’t get disrupted. 

As a project manager, I understand and foresee the project risks./As a team 
member, I observe that the project manager understands and foresees the project 
risks. 
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REQUIREMENTS MANAGEMENT Section (27 Questions — About 5-9 minutes) 


RM1. Check the statement/s that applies to the project. (Check all that apply.) 

L_] There is a requirements development document (how they are gathered and developed). 
L_] There is a requirements management document (how they are handled). 

[_] There is an agreed/negotiated requirements baseline. 

L_] There is a requirements baseline document and it is managed. 

[_] None 


RM2. Check the statement/s that applies to the project. (Check all that apply.) 

[_] Oral requirements are used. 

L_] Written requirements are used. 

[_] Requirements are formal — a standard guides the development; have identifiers and traceability matrix etc. 
[_] Requirements are informal — requirements are just identified and listed. 


[_] None 

RM3. Which of the following activities are conducted in the project? (Check all that apply.) 

L_] Market surveys [_] Customer/User interviews — [_] Prototyping [_] Scenarios/ use cases [_] Observation of the user in 
operation [_] None 


RM4. Check the statement/s that applies to the project. (Check all that apply.) 

[_] Stakeholders are identified prior to requirements development activities. 

L_] Requirements related documents have versions. 

[_] There is a requirements traceability matrix (or a similar document to trace the requirements during all the development activities). 

L_] Requirements volatility (number of requirements change/ percent of number of requirements change etc.) metrics are collected and used. 
[_] Testing team is involved in the requirement development activities. 

[_] None 


(5) (4) (3) (2) (1) N/A 


Completely Agree Neutral Disagree Completely Not 
RMS | Requirements prioritization is conducted and used for development decisions. fal Z Z a ae a 

Completely Agree Neutral Disagree Completely Not 
RM6 | All stakeholders are involved in the requirements development. ial = q = i ae ae 





Completely Agree Neutral Disagree Completely Not 
RM7 | Users or user representatives are involved in the requirements development. al Z = = ar ar 
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i Sas A : Completely Agree Neutral Disagree Completely Not 

RM8 Stakeholders show commitment to requirements stability during the project Moree Disagres Applicable 
development. CT O O O CI C1 
Completely Agree Neutral Disagree Completely Not 

Automated requirements development and management tools are used. Ty Z = 7 TL aie 


Completely Agree Neutral Disagree Completely Not 

All requirements are traceable. ial = Z z ai ci 
° ° ° pe Completel Agree Neutral Disagree Complete! Not 

Product components and project deliverables can be mapped to specific eas : : i Disagtes’ Applicable 
requirements. C] O O O CI CI 


Completely Agree Neutral Disagree Completely Not 
RM12 | Requirements are clear / unambiguous. fete pisaglee:- ||) <phlcanle 


Completely Agree Neutral Disagree Completely Not 
Requirements are complete. ‘al q Z = “ smi 
Completely Agree Neutral Disagree Completely Not 
There are no inconsistencies among requirements. Ty] Z = = sail ‘or 











: : : : ; letel A Neutral Di letel N 
During the project development, requirements related issues are resolved with the Ser : cia cee Eee eae Applicable 
negotiation with the customers. [| [| [ [ 
Completely Agree Neutral Disagree Completely Not 
RM16 | Requirements are validated with the user, customer and necessary stakeholders. Ty a Z = Pisegiee:~ |} | saPRleavle 
: ; . : Completel Di letel Not 
There are designated points of contact (people) representing various stakeholders pens . acai aa Réplicabis 
to resolve requirements related issues. [| a 
: - anat Completel A Neutral Di Completel Not 
Riig | The procedures are formal for requirements validation (what the customer pr ecies sis scsi eeer Wi peeaes” Il Meaicabie 


want). L] LJ ea] | [ FE F 


: sue : N 
The procedures are formal for requirements verification (the system does | “xmcr” | “3° ve me tee | GA sie able 
what requirements state). [] ] [] [] ie] [ 
Completely Agree Neutral Disagree Completely Not 
There is a formal requirements change procedure and document. Ty] = = = cab iil 


Completely Completely Not 


Requirements history and rationale for requirements changes are ( Dissaiee Applicable 





recorded. L 
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‘ i . 5 | N 
Requirements are worded simple and each requirement consists of only | “res” | = “°° mean cee ee Ul ieegea? Ut aeicanis 
one concept. [] CI CI CI [| CI 

Completely Agree Neutral Disagree Completely Not 
Extra effort is spent to make the requirements testable. imi = = 7 TL aie 


There are testing plans to check if the requirements are implemented as | °°.” ae cial rarer” Wt ase ||" saeaie 

intended. C] [] fil [] [] C] 
Completely Agree Neutral Disagree Completely Not 

User/customer profiles are identified and documented. im = = a ee Sar 


: . | Di letel N 
RMo6 | Requirements are constantly changing and all changes are being a ia naar ee Baa rica 
implemented. 


— = Neutral = te Not 
RM27 | Requirements are kept stable at some point. ial bei ee 


STAKEHOLDER INVOLVEMENT Section (12-16 Questions — Abe ~ About - tS 7 nutes) 








| da) 

° ° ° ° oe ) =m ree ms = ree Complete! cS 
Various users and/or customers are involved in the requirements development and = ; ? : near ag 
functionality/feature identification process. 

Vv d/ t ‘fied dd ted for th ane = — aoe —- = 
arious user and/or customer concerns are specified and documented for the - — bail 


project and the product. 


— -_ oe an? =ee 7 

Various user and/or customer profiles are identified and documented. ical ee ae 
ane el _ aes aes 7 

SI4 (ial vail ur 
° sy ° oe ce ° ° C me = ree am = ree =e letel s 

Executive/upper management is involved in the decision making process regarding = j : : pea pleas 

the project baselines, cost and schedule variations etc. 

a -_ am — au s 

All stakeholders are identified and documented. ial ial cum 
a zl am — =a 7 

There are regular meetings with various stakeholders. il ie ae 


Prototypes/user stories/paper mock-ups/use cases etc. are prepared with the 
involvement of users. 





There is an information gathering activity to identify stakeholders and | “°;Ps°” aor ia mec ll ates ll esas 
their stakes/concerns. [] | i L] x a 





All stakeholders show commitment to the successful outcome of the | “'r..” acd mi Bree ll apteres ||| Ae aicaie 
project. L | il L | [| CI CI 


$l10. Check the statement/s that applies to the project. (Check all that apply.) 

[_] There is a document guiding the management of stakeholders. 

L_] The stakeholder management plan/document lists the primary and secondary stakeholders. 

L_] The stakeholder management plan/document lists the concerns and stakes of the primary and secondary stakeholders. 
[_] The stakeholder management plan/document provides specific strategies for dealing with various stakeholders. 

[_] The users and/or customers participated in the testing phase of the project. 

[_] There is a documented procedure for the acceptance of the project deliverables. 





[_] None 
* Respond to the following questions(SI11-SI12) only if the project is developed for the market without a specific contract. 
Gj41 | The marketing department and necessary functional managers are involved in the a ee Derg re Blea nt 


decision-making process during development. 


Completel Agree Neutral Disagree aie letel Not 
The marketing department provides timely information regarding users and other = y : . nase nice 
competing products. 


Respond to the following questions (SI13-SI18) only if the project is developed under a contract with a specific customer. 


There are communication and coordination problems between project | “rss” | “se pea Tea Wee | erie 
team members and other stakeholders. iL] C] C [] [ L] 
When there is a change in the baseline, the cost, schedule, and| “rr” | “3° eae (aes a We ee 
functionality/features are renegotiated with the customer. [| [ L] L] L] L 





° ° ° Completel Agree Neutral Disagree Completel Not 
Regular updates regarding project variables such as cost, schedule and progress on pees 2 . Digsarer | Japeucepié 
functionality are provided to the stakeholders. [| [| [| [| [| [| 
° ° ° ° Completel Agree Neutral Disagree Completel Not 
When there is an increase in cost or delay in schedule, the news and the Veics : i i nasa ‘orto 
consequences are shared with the stakeholders in a timely manner. [| [| [| [| 


° ° ° ° Completel Agree Neutral Disagree _ letel Not 
Project milestones are considered reached when there is consensus from asics ; : : nee ‘orto 


stakeholders for advancing to the next phase. [| [| L] L] 


$I18. Check the statement that applies to the project. (Check only one.) 
[_] Project team members are allowed to have direct communication with the customers and/or users. 
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[_] All communication with the stakeholders is conducted via the project manager and/or management. 


PROJECT MONITORING AND CONTROL Section (19 Questions — About 4-8 minutes) 


PMC1. Check the statement that applies to the project. (Check only one.) 
[_] There is a documented project plan. [_] There is no project plan. 


PMC2. Which of the following data and/or metric/s are regularly monitored and documented? (Check all that apply.) 
[_] Team/developer performance 

[_] Cost and earned value 

[_] Risk items and their impacts 

[_] Schedule performance 

L_] Number of requirements changes 

L_] Necessary staff and skill requirements 

[_] None 


PMC3. Check the statement that applies to the project. (Check only one.) 
[_] There are specific project team members assigned for controlling activities such as configuration management, requirement changes etc. 
[_] All control activities are handled by the project manager. 


PMC4. Check the statement/s that applies to the project. (Check all that apply.) 

[_] There are project progress or milestone review meetings. 

L_] Key project problems are identified and being monitored. 

[_] Key project problems and project progress status is visible to the stakeholders including project team members. 
[_] None 


PMC5. Check the statement/s that applies to the project. (Check all that apply.) 

[_] There is an established requirements change and control process. 

[_] There is an established risk management and control process. 

[_] There is an established configuration management process. 

L_] There is an established baseline tracking and scope change control process. 

_] There is an established project management data and metrics collection and monitoring process. 
[_] None 
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PMC 
10 
PMC 
11 
PMC 
12 
PMC 
13 
PMC 
14 
PMC 
15 
PMC 
16 
PMC 
17 
PMC 
18 
PMC 
19 


The project problems are generally proactively addressed (before they happen). 


The project problems are generally reactively addressed (when they happen). 


The project resources are closely monitored. 
There is an established project monitoring and control procedure with the 
acceptance of project team members. 


There are established methods/criteria to determine deviations from the project 
plan. 


In case of deviations from the plan, corrective action is immediately taken. 


Project management metrics are effectively collected and used in decision-making. 
(such as planned versus actual cost, requirements changes, schedule performance 


A project management automated software tool is used to manage project 
management data and metrics. 


Earned value management is effectively used. 


There is communication between management and project staff regarding the 
project progress data. 


The commitment and concerns of various stakeholders is being monitored through 
regular meetings and communication. 


The subcontractor performance is monitored regularly. 


There are checklists for critical tasks such testing, version control, requirements 
change requests etc. 


Corrective actions for problems are timely and effective. 
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PROJECT PLANNING AND ESTIMATION Section (35 Questions — About 10-18 minutes) 


PPE1. Check the statement/s that applies to the project. (Check all that apply.) 
L_] There is a formal documented project plan. 

L_] There is an informal project plan. 

L_] There project plan and schedule is made visual via diagrams, charts etc. 

L_] There is no project plan. 


PPE2. Check the statement that applies to the project. (Check only one.) 
[_] The project plan is developed as needed during the project. [_] The project plan is developed up front before any development effort. 


PPE3. Check the statement that applies to the project. (Check only one.) 
L_] The project budget, schedule, and staff requirements are strictly enforced by the executive/upper management or customer. 
L_] The project budget, schedule, and staff requirements are identified via analysis and negotiation. 


PPE4. Check the statement that applies to the project. (Check only one.) 
L_] The project plan is approved by the stakeholders such as customers, users, project team members, executive management etc. 
[_] There is no approval process. 


PPE5. Which of the following/s is/are involved in the project planning? (Check all that apply.) 
L_] Senior/executive/upper management 

L_] Experts and consultants 

[_] Project manager and/or management team 

[_] Project team members 

L_] Customer/user/marketing department 

[_] Other relevant stakeholders 

[_] None 


PPE6. Which of the following/s is/are included in the project plan? (Check all that apply.) 
L_] Project scope 

[_] Deliverables or products list 

|_] Detailed schedule and milestones / various product version delivery dates 

L_] Detailed budget and cost analysis 

L_] Staffing/personnel/developer requirements 

[_] Task responsibility matrix or similar assignment matrix 

[_] Required functionality/features of the products or deliverables 
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[_] Validation and verification plan 

[_] Acquisition plan / Subcontracting planning 

[_] Deployment or Installation plan/ Marketing plan 
L_] Quality requirements / Quality assurance plan 

[_] Risk management planning 

L_] Project glossary 

L_] Project communications planning 

L_] Project organization charts 

L_] Staff responsibilities and responsibility definitions 
L_] Necessary facility, equipment, and component requirements 
[_] None 


PPE7. Check the statement/s that applies to the project. (Check all that apply.) 

L_] There is a statement of work (or a similar document) stating what needs to be accomplished/done. 

[_] There is a work breakdown structure or a feature/functionality list (or a similar document) that details the project tasks/activities. 
L_] The tasks and activities are identified as the project progresses. 

[_] None 


PPES8. What kinds of effort, schedule or cost estimation techniques are used? (Check all that apply.) 
L_] Experiences of project manager/management team 

L_] Inputs from project team members 

L_] Expert or consultant judgment 

L_] Analogy to similar projects 

L_] Historical data 

[_] Automated cost estimation tools 

[_] None 


PPE9. Check the statement/s that applies to the project. (Check all that apply.) 
[_] No estimation is needed. 
[_] Only one type of estimation technique is used. 
Two or more estimation techniques are used. 
= Estimates from various techniques are compared and analyzed for discrepancies. 
None 


PPE10. Check the statement/s that applies to the project. (Check all that apply.) 


L_] Lines of code (LOC) are used in estimation. 
L_] Function points are used in estimation. 
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[_] Number of functionality/features are used in estimation. 

[_] Number of modules and deliverables are used in estimation. 
[_] Other advanced metrics used in estimation. 

[_] None 


PPE 


ay 
jo 


PPE 
12 
PPE 
13 
PPE 
14 
PPE 


PPE 
18 


—_ 
N 


(5) (4) (3) (2) (1) N/A 


Completely Agree Neutral Disagree Completely Not 
The project schedule is feasible. ial q Z = eT ne 
Completely Agree Neutral Disagree Completely Not 
The funding for the project is adequate. ial q Z Z cae isi 
Completely Agree Neutral Disagree Completely Not 
The project is adequately staffed. ial 7 = = in ne 
Completely Agree Neutral Disagree Completely Not 
Extra funding for unprecedented issues is set aside. ial = = = ae in 
Completely Agree Neutral Disagree Completely Not 
Slack or buffer time exists in the schedule for unprecedented or extra activities. ial = Z = vn ia 
Alternative staff to accomplish critical tasks/activities are considered and aes ore or ae ie Applicable 
incorporated in the project plan. [| [| [| [| [| [| 
Completely Agree Neutral Disagree Completely Not 
All relevant stakeholders are identified before planning activities. Ty] TJ Z = eee na 
A certain level of requirements analysis is conducted before planning and aren ae sical nee eee Agaiicakie 
estimation. [| [| [| [| [| [| 
All external dependencies are identified and incorporated to the planning. (Such as Completely Agree Neutral Disagree Completely Not 
acquisition of various products and services from outside vendors, required ial = Z Z ae sc 
permissions from various authorities, etc.) 
Completely Agree Neutral Disagree Completely Not 
The project plan is updated throughout the project development. ‘al a a = ey iat 
ao eanne : Di letel N 
The project plan is visible/available to project team members and other | “rer” | “sree waa eae Nae | eat ple 
relevant stakeholders. C] CI] C] a CJ [] 
Di letel N 
Various automated project management tools are used in planning the | “ec” noe el ee eee |r le 
project. C1 a O o C] C1 
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: z ‘ = = C letel A Neutral Di Col letel N t 
The project team members are consulted in planning and estimation 
efforts. LI 
. | | h . | . d . . Completely = _ ens —— = 
The managers at various levels have project planning and estimation oe — Applicable 
training. a L] 
= age . “gs . C letel im / tral C LC tel Not 
Each task/activities/work packages are assigned to specific project team | ~°x!... ” one Are a mesa Applesbie 
member or members. [] L] 
Completely — a oz _—— Not 
Critical activities are identified and/or critical path analysis is conducted. ial E i eal al 
. 2 2 kI . d | d ace = ree oe Disagree Completely Not 
Various standards, guidelines or checklists are used in planning an — Dissorse Applicable 
estimation. [J [ie [ 
C letel = = tral Di C letel Not 
Formal analysis is conducted for cost, schedule and effort estimation such = ao ie as picsuiee. |. “pplicabie 
as PERT, CPM etc. [ee le [el L] 
letel A a | Di C letel Not 
Factors such as staff turnover or loss of key personnel are considered Sa aed me ee picauiee. || “Apalicabie 
during planning. J CJ [| [vl [| 
Completely Agree = Disagree Completely Not 
Realistic estimates guide the project planning. Here picagieg:. | -nppiceve 
L_] L_] L] L_] 
Completely mh Neutral Disagree Completely Not 
Testing is carefully incorporated to project plan. int ana sn 
Completely a = eee as = 
Effort estimations are provided by those performing the tasks. i q ae i Ui 
. “ . " " " Completely Agree _ =e _— = 
Project risks are carefully analyzed and contingencies are included in the ee aa fant 
planning. [| [| 
: : : s ope . C letel A = | - C a tel LI t 
A suitable project development approach and process is identified with cca ic eis pauses pisaiee: 4). cmpalicebie 
rationale in the project plan. Rul [J -¥ [if [| 
aa Agree Neutral Disagree Completely Not 
All necessary skills and expertise needed in the project are identified. ial q 9 ich oa 
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SCOPE MANAGEMENT Section (16 Questions — About 3-8 minutes) 


SM1. Check the statement that applies to the project. (Check only one.) 
[_] Project scope never changed. [ ] Project scope frequently changed. _[_] Project scope somewhat changed. 


SM2. Check the statement that applies to the project. (Check only one.) 

L_] Project scope is ambiguous at first and it becomes clear during the project. 

[_] Project scope is ambiguous at first and stays ambiguous due to various reasons. 

[_] Project scope is defined and clear at the beginning of the project and it stays clear. 

L_] Project scope is defined and clear at the beginning of the project and it become ambiguous due to various reasons. 


SM3. Check the statement that applies to the project. (Check only one.) 

[_] There is a project scope document and it stayed the same from the project start. 
[_] There is a project scope document and it is updated when it is necessary. 

L_] There isn’t a project scope document. 


SM4. What is the effect of project scope changes on the project schedule? (Check only one.) 
L_] None L_] On time without scope change/s L_] On time with scope change/s [_] Late without scope change/s 
L_] Late with scope change/s 


SM5. What is the effect of project scope changes on the project budget? (Check only one.) 
[_] None 

L_] Within budget without scope change/s 

L_] Within budget with scope change/s 

L_] Cost overrun without scope change/s 

L_] Cost overrun with scope change/s 


SM6. What is the effect of project scope changes on the functionality of the deliverables? (Check only one.) 
[_] None 

[_] Full functionality without scope change/s 

L_] Full functionality with scope change/s 

[_] Less than planned functionality without scope change/s 

[_] Less than planned functionality with scope change/s 


SM7. Check the statement/s that applies to the project. (Check all that apply.) 


L_] Project scope changes are handled only by the management. 
_] Project scope changes have to follow a formal defined process. 
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[_] Project scope changes follow a decision-making process that includes management, stakeholders, and team members. 
[_] Project scope changes handled informally by the management. 


SM8. Which of the following statement/s is/are included in the project scope document, if there is one. (Check all that apply.) 
[_] The problem statement 

[_] The work to be done or work breakdown structure 

[_] The constraints 

[_] The resources 

[_] Preliminary or detailed schedule and cost analysis 

L_] The project deliverables 

L_] Clear definition of performance to meet contractual and legal obligations 

L_] Glossary 

[_] Not Available 


SM9. Check the statement/s that applies to the project. (Check all that apply.) 

L_] The project scope is defined after stakeholders are identified. 

L_] There is at least one project scope identification/definition meeting at the beginning of the project. 
L_] There is a project scope change board. 


SM10. Who are included while defining and updating the project scope? (Check all that apply.) 
L_] Project management team 

L_] Project manager 

[_] All stakeholders 

[_] Some stakeholders 

[_] Project team members 

[_] Subcontractor representatives if there is subcontracting 

[_] None 
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SM11 


SM12 


SM13 


SM14 


Before defining the project scope, there is a rigorous information gathering | completely Agree Neutral Disagree | Completely Not 
activity about the problem that is to be solved, the resources, the fal ae a 
constraints, the deliverables etc. 

Completely Agree Neutral Disagree Completely Not 
Project scope is not clearly defined due to various reasons. ml Said ais 
The project has a documented project scope definition and a formal scope a 7 — _ es r 

i te a 

change process. 

° ° ee ° a 
Project scope is always visible and clear to stakeholders, project team members, a = — = ae wt 
and management. 

= ms =e = _— = 
Project scope changes have to go through an extensive decision-making process. in = = tee a 

— Agree = Disagree Completely Not 
The project scope document is reviewed and approved by all stakeholders. at Z = Gai i 





RISK CONTROL Section (17 Questions — About 3-8 minutes) 


RC1. What is the overall risk level of the project? (Check only one.) 
L_] High [| Medium [_] Low [ ] None 


RC2. What is the effect of risks on the project budget? (Check only one.) 
[_] High cost overrun [|] Medium cost overrun[L_] Low cost overrun [_] None 


RC3. What is the effect of risks on the project schedule? (Check only one.) 
L_] The project delivery is on time. [_] The project delivery is slightly late. | [_] The project delivery is significantly late. 


RC4. What is the effect of risks on the project functionality? (Check only one.) 
L_] High [-] Medium L_] Low [_] None 


RC5. What is the level of funding and resources set aside for risk management? (Check only one.) 
[_] More than enough  [_] Enough L_] Hardly enough L_] No funding and resources 


RC6. Check the statement/s that applies to the project. (Check only one.) 
[_] Adequate slack time is planned in the schedule for consequences due to risks. 
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[_] There is not any slack time planned for consequences due to risks. 

[_] Not enough slack time is planned in the schedule for consequences due to risks. 

RC7. Check the statement that applies to the project. (Check only one.) 

[_] Risks are handled when they occur. [_] Risks are addressed before they occur. L_] Both 


RC8. Check the statement/s that applies to the project. (Check all that apply.) 
L_] Informal project risk management procedures are in place. 

[_] Project risk management is based on formal procedures. 

L_] There is not any project risk management and planning. 


RC9. Check the statement/s that applies to the project. (Check all that apply.) 

[_] Risks are generally avoided. (Risk Avoidance) 

L_] Risks are transferred to third parties for example contracting risky development items to consultants or experts. (Risk Transfer) 

[_] Risks are managed as they occur. 

L_] Risk mitigation (actions reducing the severity/impact of a risk) is the most used option in risk management of the project. (Risk Mitigation) 
[_] None 


RC10. Check the statement that applies to the project. (Check only one.) 

L_] Experts are consulted in the risk management of the project. 

[_] Project management handles all the risks. 

[_] Project team members and stakeholders are involved in the risk management. 


(1) 


Completely Agree Neutral Disagree Completely Not 
RC11 | For each identified risk item, there is an information gathering activity. fink = = Z an iu 
RCL, | Contingencies and alternative solutions are planned for the critical tasks ae || are mega! || tomes | Nh eee 
and portions of the development exposed to high risks. C] CJ C] CI] [J 1] 
Completely Agree Neutral Disagree Completely Not 
RC13 | Top risk items list is closely monitored and periodically updated. ial Z q ae ni 
Completely Agree Neutral Disagree Completely Not 
RC14 | Risk monitoring is an important activity in the project. i z Z = ae ni 
Completely Agree Neutral Disagree Completely Not 
Agree Disagree Applicable 





L 






RC15 | Risk avoidance is primary method of risk control activities. 
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RCI Minar 
RC16 : ; Agree Disagree Applicable 
handled through project status meetings etc. [] I CI 1 [| [1] 
Completely Agree Neutral Disagree Completely Not 
RC17 | There is a risk management plan and course of action for each high-risk items. T] Z oi aie 


STAFFING/HIRING Section (29 Questions — About 7-13 minutes) 


$1. Which of the followings are clearly identified, documented and communicated? (Check all that apply.) 
L_] Project Roles L_] Project Positions L_] Necessary Qualifications for the project [_] None 


$2. Which of the documents or similar documents exist for the project? (Check all that apply.) 
L_] Project staffing management plan 

[_] Project responsibility/accountability/interfaces/assignment matrix 

L_] Project work breakdown structure 

[_] None 


$3. What is the experienced-to-inexperienced project team member ratio? (experienced: inexperienced) (Check only one.) 
[]Smallerthan1:2 [] 1:2 L] 4:1 [J 2:1 L_] Greater than 2:1 


$4. Which of the followings for team members are clearly identified, documented and communicated? (Check all that apply.) 
L_] Responsibility L_] Job Interfaces L_] Reporting Structure [_] None 


ss | The work breakdown structure (WBS) or similar document is completed ee |i “eee era meee Ill ipeeareee |) Aeie ails 
before hiring/staffing. C C] C] C] C] a 
Completely Agree Neutral Disagree Completely Not 
S6 The analysis of the required work and resources is conducted rigorously. ial Z Z Z a cai aie 
Completely Agree Neutral Disagree Completely Not 
S7 Significant project risks are identified before the hiring/staffing the team. ial Z Z = a ail wee 












Completely Agree Neutral Disagree Completely Not 
There is adequate funding and resources for hiring/staffing. ia 9 5 5 meee oe 
Th d t k f d t ith th kill d ti Completely Agree Neutral Disagree Completely Not 
ere are adequate work force and experts wl e necessary skills and expertise oie piesgres Apsicsbis 
available for hiring and/or staffing on this project. [| ra H T H oH 
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Expertise on human resources is acquired for staffing and hiring activities. 
Project open positions are made attractive to qualified candidates through 
incentives etc. The position is made desirable. 


The skills and expertise needed for the project success are acquired with the timely 
recruitment of team members. 


The necessary interpersonal skills for the roles are identified and the project team 
members are recruited also based on their interpersonal skills. 


The ambitions and goals of the project team members are aligned with the project 
mission and goals. 


The project team members have the necessary educational background. 


The project team members have similar project work experience. 


The productivity of the project team members are within the expectations. 
Project team members are familiar and comfortable with the organizational 
culture. 


Project team members have _ difficulties with 
procedures. 


the organizational 


Project team members are happy with their roles, positions and career 
advancement opportunities in the project organization. 


Project team members stay with the project according to the project staffing 
management plan. Turn-over rate is at minimum. 


Resignations are at minimum. 


Project team members acquire the necessary skills and expertise needed for the 
project through training and coaching. 
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L_] 
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Neutral 


Neutral 


Neutral 


Neutral 


Neutral 


Neutral 


Neutral 


Neutral 


Neutral 


Neutral 


Neutral 


Neutral 


Neutral 


L] 
L] 
L] 
L] 
L_] 
L_] 
L_] 
L_] 
L_] 
L_] 
LI 
LI] 
L_] 
LI] 


Completely 
Disagree 


Completely 
Disagree 


Completely 
Disagree 


Completely 
Disagree 


Completely 
Disagree 


Completely 
Disagree 


Completely 
Disagree 


Completely 
Disagree 


Completely 
Disagree 


Completely 
Disagree 


Completely 
Disagree 


Completely 
Disagree 


Completely 
Disagree 


Completely 
Disagree 


Not 
Applicable 


Not 
Applicable 


Not 
Applicable 


Not 
Applicable 


Not 
Applicable 


Not 
Applicable 


Not 
Applicable 


Not 
Applicable 


Not 
Applicable 


Not 
Applicable 


Not 
Applicable 


Not 
Applicable 


Not 
Applicable 


Not 
Applicable 





There are alternative team members with the necessary skills and knowledge to Completely Agree Neutral Disagree Completely Not 
take over some other team member’s work for critical tasks in case of team gice q = Z Pisagree said 
member loss. 
Completely Completely Not 
Agree Disagree Applicable 
O 
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Completely Completely Not 
Agree Disagree Applicable 


Completely Completely Not 
Agree Disagree Applicable 





Completely Agree Neutral Disagree Completely Not 
When necessary, consultants and contractors are used effectively. Ty] = Z = pisagiee ine 


CONFIGURATION MANAGEMENT Section (13 Questions — About 3-7 minutes) 


* In some organizations, configuration management is referred to as version control. 





CM1. Check the statement that applies to the project. (Check only one.) 
L_] Configuration management is conducted informally. 
L_] Configuration management is a formal and documented activity and it has well-defined procedures. 


CM2. Check the statement/s that applies to the project. (Check all that apply.) 
L_] There is a configuration management document. 

[_] There is a configuration or change control board, committee or team. 

L_] There is a configuration items list. 

[_] None 


CM3. Check the statement/s that applies to the project. (Check all that apply.) 

|_] Baselines and configuration items are identified at the beginning of the project and updated as necessary. 

[_] The owner or responsible staff is identified for each configuration item. 

[_] Every configuration item has a unique identifier. 

L_] Important characteristics for each configuration item are identified such as author, type, date, version number etc. 
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[_] None 


CM4. Check the statement/s that applies to the project. (Check all that apply.) 

L_] The configuration management procedures includes a detailed change and change request protocols. 

[_] The configuration management system has various levels of control (such as only author may release the item, restricted write access etc). 

L_] There is not a configuration management system and configuration management is only the responsibility of project team members or 
developers. 

[_] None 

CM5. Check the statement that applies to the project. (Check only one.) 

L_] The change requests have to go through the change control board or responsible staff. 

[_] The change requests are only handled by the developer or the owner of the configuration item. 


CM6 
M7 


C 


CM8 


CM 


CM10 


CM11 


CM12 


CM13 


Completely Agree Neutral Disagree Completely Not 

The project suffers from configuration/version management problems. Ty] Z Z Z re sini 
. letel Neutral Di letel N 

An automated configuration management system is used and adequate for ae || ee fou Roe Nee |: Sa Be 
the project. i] [J [ [| [| [ 
The configuration management procedures are strictly followed. Project team pea a pena pee. Wee: | abe 
members do not try to bypass them. ii [J [ [ ] [| [ 

Completely Agree Neutral Disagree Completely Not 
The integrity, security and privacy of configuration items are satisfactory. Ty] = = = em in 
The changes and change requests are controlled, and documented in such a way eas aes Dae: eg eee ice 
that it enables audit. [| [| [] [| [| [| 

Completely Agree Neutral Disagree Completely Not 
Every change request is controlled and extensively reviewed. Ty] Z = Z ae imi 
Records of configuration management activities, changes to baselines, work aa ans ae: eg eee ice 
products, and change requests are well-maintained. [| [| [| [| [| [| 


° ° ° A ° ° ° Completel Agree Neutral Disagree Completel Not 
There is an established and reliable configuration management system including ee : = Bigsarees | IRBoIcsbI 
automated tools, databases, protocols etc. [ [] [] [J [J [ 
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RISK ASSESSMENT Section (20 Questions — About 5—10 minutes) 


RA1. Which of the following does best characterize the risk assessment activities in the project? (Check only one.) 
L_] Formal L_] Informal [_] Semiformal [_] Not available 


RA2. Check the statement/s that applies to the project. (Check all that apply.) 
L_] Risks are assessed as they are identified during the project. 

[_] Risks are assessed early and incorporated into a risk management document. 
[_] The risk management document is periodically updated. 

_] There is staff specifically assigned to risk assessment activities. 

[_] Lessons learned are visited prior to risk assessment activities. 


[_] None 

RA3. In which of the following categories the risks are assessed and documented? (Check all that apply.) 

L_] People [_] Schedule [| Budget and Funding [_] Technology [_] Requirements L_] Subcontractor 
None 


RA4. There are common objective criteria to assess risks. (Check only one.) 
[ ] Yes [ ] No L_] Partially [_] Not Available 


RA5. Check the statement/s that applies to the project. (Check all that apply.) 

L_] There is a project risk management plan. 

L_] The project risk management plan includes objective criteria for risk identification, analysis and prioritization. 
[_] Project risk document is updated frequently along the project. 

[_] None 


RA6. Check the statement/s that applies to the project. (Check all that apply.) 
L_] Experts or consultants are used for risk assessment. 

[_] Experienced project staff is used for risk assessment. 

L_] Project manager conducted the risk assessment. 

[_] There is not any risk assessment activity. 


RA7. Check the statement/s that applies to the project. (Check all that apply.) 
L_] Risks are identified. |_] Risks are analyzed. [_] Risks are categorized. [_] Risks are prioritized. [_] None 


RA8. Check the statement that applies to the project. (Check only one.) 
[_] Risk assessment is based on qualitative methods. 
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L] 


[_] Risk assessment is based on quantitative methods. 

[_] Risk assessment is based on the judgment of the management. 

[_] Risk assessment is based on both qualitative and quantitative methods. 
L_] There is no need for any risk assessment activity. 


RA9 


RA10 


RAI1 


RA12 


RA17 


RA18 


The projects risks are documented early with details related to their impact 
on the project. 


Risk assessment has a clear impact on project planning and decisions. 


Sufficient reserve resources and funding are planned and set aside for risk 
assessment activities 


Top risk items list or a similar list is maintained. 





Risks are assessed with the broad inclusion of stakeholders and project team 
members. 


Project environment facilitates and encourages open and free discussions on project 
risks. 


Risks are identified using risk identification tools such as checklists, databases, risk 





taxonomy, decision-driver analysis, etc. 


Risks are analyzed based on their probability of occurrence and impact on the 
project. 

Risks are prioritized based on their probability of occurrence and impact on the 
project. 


Risk assessment information is always visible and they are shared with stakeholders 





and project team members. 


Any stakeholder or project team member may report a risk at any time and there is 





a mechanism allowing such reports. 
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Completely 
Agree 


L 


Completely 
Agree 


LJ 


Completely 
Agree 


L 


Completely 
Agree 


LJ 


Completely 
Agree 


LJ 


Completely 
Agree 


L] 


Completely 
Agree 


L 


Completely 
Agree 


L] 


Completely 
Agree 


L] 


Completely 
Agree 


LJ 


Completely 
Agree 


L] 


Agree Neutral Disagree Completely Not 
Disagree Applicable 

Agree Neutral Disagree Completely Not 
Disagree Applicable 

Agree Neutral Disagree Completely Not 
Disagree Applicable 

Agree Neutral Disagree Completely Not 
Disagree Applicable 

Agree Neutral Disagree Completely Not 
Disagree Applicable 

Agree Neutral Disagree Completely Not 
Disagree Applicable 

Agree Neutral Disagree Completely Not 
Disagree Applicable 

Agree Neutral Disagree Completely Not 
Disagree Applicable 

Agree Neutral Disagree Completely Not 
Disagree Applicable 

Agree Neutral Disagree Completely Not 
Disagree Applicable 

Neutral Disagree Completely Not 
Disagree Applicable 


> 
Oo? 


RA20. (Answer only if a portion of the system is subcontracted.) Check the statement/s that applies to the project. (Check all that apply.) 
[_] Subcontractor/s is/are free in their risk management decision and activities. 

L_] Subcontractor/s is/are contractually responsible to have formal risk assessment procedures. 

[_] Subcontractor/s is/are contractually responsible to deliver risk assessment reports. 

[_] Subcontractor/s has/have a representative for project risk management meetings. 


QUALITY ENGINEERING Section (20 Questions — About 4-10 minutes) 


QE1. Check the statement/s that applies to the project. (Check all that apply.) 
[_] There is a quality policy. 

L_] Quality is not a high priority in this project due to various reasons. 

L_] There is a quality planning activity. 


QE2. Check the statement/s that applies to the project. (Check all that apply.) 

L_] Quality expectations of various stakeholders are identified and documented. 

[_] The quality standards and guidelines related to the project are identified. (Such as aviation standards etc.) 
[_] Objective quality criteria for the project and its deliverables are identified. 


[_] None 
QE3. Which of the following quality attribute/s are considered achieved in the project? (Check all that apply.) 
L_] Maintainability L_] Safety L_] Security L_] Reliability [| Usability | [_] Other [L_] None 
QE4. What is the amount of testing conducted during the project development? (Check only one.) 
[ ] Extensive  [_] Fair [ |] Some [ ] None 
(5) (4) (3) (2) (1) N/A 
Completely Agree Neutral Disagree Completely Not 
QE5 | Quality is considered a high priority in this project. igi eye aie 
= Agree Neutral Disagree cet RS 
4 There is support for and commitment to quality from executive management. i iketiedied C3 
= Agree Neutral Disagree = 
High quality is planned from the start in this project. Ty) — — — ae pra 


a= Agree Neutral Disagree = a 
Various quality metrics are identified. ‘ai — - - eae i 
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Completely Agree Neutral Disagree Completely Not 
Quality assurance procedures are adequate. ‘iat a aie 
a= ml oh “— = — 
Quality assurance procedures are documented. Ty] = = S ecm ae 
= Agree Neutral Disagree Completely Not 
Adequate amount of resources are set aside for quality engineering activities. Ty] = ele a 
= Agree Neutral Disagree Completely Not 
The requirements are defined with the guidance of quality expectations. Ty] = eee: el 


= 
 Qe13 The project team culture encourages commitment to high quality. = 
a= Disagree = a 
Project team members are trained in quality assurance. cai z ‘aie i ti 
a= al Disagree = — 
There are quality thresholds and expectations for various work products such as ay aca nea 2 
system architecture, requirements definitions, designs, testing etc. i 
a= utral Disagree = — 
Quality considerations are limited to testing. Ty] iz eye aie 
= Agree Neutral Disagree = = 
QE17 | High testing coverage for the product is achieved. Ty] Te ine 


tl mI tL 5b =a CI 
QE18 | There are adequate tools, equipment, and resources for testing. ial eae ele 


= +t amr a5 = = 
There are specifically assigned team members for quality related issues. ] sick ae 


QE20. Which of the following activity or activities are conducted during the project ee (Check * that Tr 
[_] Design reviews 

L_] Code reviews/inspections 

L_] Performance testing 

L_] Independent verification and validation 
L_] Quality assurance activities 

[_] Requirements tracing 

L_] Various types of testing 

[_] Defect identification and prevention 

[_] Simulations and/or prototyping 

[_] None 
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Thank you very much for your time and participation. After completing the questionnaire, please respond the 
following question again. It does not have to be the same as your initial assessment. 


After you completed the questionnaire, how would you rate the overall success of the project? 
(0 being a complete failure and 10 being a complete success.) 


PR18. 
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EVALUATION INSTRUMENT SCORES 
SA 


APPENDIX C. SOFTWARE PROJECT MANAGEMENT 
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APPENDIX D. SOFTWARE PROJECT MANAGEMENT 
EVALUATION MODEL IN DETAIL 


TABLE OF QUESTIONS AND SCORES USED TO ESTABLISH SCALING 

AND SHIFTING FACTORS 

Sub Project Number of Lowest alte latexcy Difference 

Management Area @]UT=%-} fe) at ambelexe) a=) Tore) c=) between 
the 
highest 
and lowest 
Tore) c=) 

Communication 23 104 

Teamwork 30 127 

Leadership 17 68 

Organizational 26 112 

Commitment 

Project Manager O/ 112 

Stakeholder 16 72 

Involvement — Contract 

Stakeholder 12 

Involvement — Market 


Staffing and Hirino 29 


Requirements 

Management 

Project Monitoring and 
Control 

Project Planning and 35 
Estimation 

Scope Management 16 
Configuration 13 
Management 

Quality Engineering 20 
Risk Assessment — No 19 
Subcontractinc 

Risk Assessment — With 20 
Subcontractinc 

Risk Control 
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SHIFTING FACTORS AND SCALING FACTORS 
S10] od nd xe) (=Leq mi r-lar-le(stan-lal@vaey:| Shifting Factor Tor: 1ifale fm at: lea cele 


Stakeholder Involvement - Contract 30 10/72 
Stakeholder Involvement - Market 10/56 
Staffing and Hiring 10/116 
Requirements Management 10/123 
Project Monitoring and Control 10/86 
Project Planning and Estimation 10/174 
Scope Management 10/71 

Configuration Management 10/60 
Quality Engineering 10/93 
Risk Assessment 10/91 

Subcontractine 

Risk Assessment — With 38 10/101 
Subcontractinc 


Risk Control 26 10/54 





162 


SUB PROJECT MANAGEMENT AREA EQUATIONS 
Communication Area Evaluation Model 
| mae 


(= 


Teamwork Area Evaluation Model 


aed t 
aScore = 7 x( ‘, Tb | 


Leadership Area Evaluation Model 

In the leadership section of SPMEI, the respondent has to choose to respond to 
one of two questions: L1/7 and L18. If the project team mostly consists of 
inexperienced staff then the respondent should answer question L17. If the 
project team mostly consists of experienced staff, then the respondent should 
answer question L18. The choices for these questions are identical. However, 
the scoring is different. The model for both cases is presented below. If the team 
mostly consists of inexperienced staff, then the leadership area model is as 


follows: 
Dw: | 
‘mL 











lf the team mostly consisted of experienced staff, then the leadership area model 
is as follows: 





Organizational Commitment Area Evaluation Model 
mage 








Project Manager Area Evaluation Model 
air 





> Pilg + 2] 


Stakeholder Involvement Area Evaluation Model 

In the stakeholder involvement section of SPMEI, the questions after SI10 are 
divided into two sections. If the project is developed for the market without a 
specific contract, then the respondent should answer questions SI11 and S112. If 
the project is developed under a contract with a customer, then the respondent 
should not answer the questions Sl11 and S112, but the questions from S113 to 
5118 instead. If the project is developed for the market, then the stakeholder 
involvement area model is as follows: 





ahkeheolider irvelpentent Area Score = 





lf the project is developed for the market, then the stakeholder involvement area 
model is as follows: 
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o fe 
takeholder heveleement Area Score = aX > } ) 


Staffing and Hiring Area Evaluation Model 


wo ( 
"Tis" (> oe 2] 








Staffing and Hiring Area Seon 








Scope Management Area Evaluation Model 
a | LEE 


zt) 
Configuration Management Area Evaluation mons 
mle 





$M; + 6] 








an é Cl; 22 
Quality Engineering Area Evaluation Model 
Titel Sh) | 
: ( On, bP | 
a mL 





Risk Assessment Area Evaluation Model 

In the risk assessment section of the SPMEIl, there is an additional question at 
the end of the section for the projects in which subcontracting is used. The 
question identifier is RA20. If the project does not utilize subcontracting, then the 
risk assessment area model is as follows: 


aY : 
RA, | 


od 1 
(> 

lf the project utilizes subcontracting, then the risk assessment area model is as 

follows: 





Aigk ae ORS SEE eae 





Risk Control Area Evaluation Mode 
In the risk control section of the SPMEI, there are four questions that are 
excluded from the evaluation model: RC1, RC2, RC3, and RC4. These questions 
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are included in the instrument to enable a consistency check among the 
responses and for other research purposes. Therefore, for the risk control area 


model, only the responses from RC5 to RC17 are included in the evaluation 
model: 





165 


THIS PAGE INTENTIONALLY LEFT BLANK 


166 


APPENDIX E. METRIC FEEDBACK INSTRUMENT 


Consent to Participate in Research 


Introduction. You recently participated in a research study entitled “The 
Effectiveness of Software Project Management Practices” conducted by the Naval 
Postgraduate School. You are now invited to provide feedback on the study. 


Procedures. The data you provided has helped assist with the development of a 
software project management effectiveness (PME) metric. The research into the 
usefulness and the applicability of this metric is ongoing. The PME metric for your 
project was calculated from the responses in your survey and provided to you. The 
purpose of this additional survey is to inquire into the usefulness and applicability 
of the original survey and the PME metric so that it can be improved in the future. 


Voluntary Nature of the Study. Your participation in this study is strictly 
voluntary. If you choose to participate you can change your mind at any time and 
withdraw from the study. You will not be penalized in any way or lose any benefits 
to which you would otherwise be entitled if you choose not to participate in this 
study or to withdraw. 


Potential Risks and Discomforts. The potential risks of participating in this 
study are: 
b) A breach of confidentiality may result in embarrassment of the research 
Subject. 


Anticipated Benefits. Anticipated benefits from this study are: 
a) To assist in the development of project management metrics and improve the 


software engineering body of knowledge to improve software project 
management; and 

b) To enable the development of a tool for you to monitor, evaluate and improve 
your projects. 


Compensation for Participation. No tangible compensation will be given. A 
copy of the research results will be available at the conclusion of the experiment. 


Confidentiality & Privacy Act. Any information that is obtained during this study 
will be kept confidential to the fullest extent permitted by law. All efforts, within 
reason, will be made to keep your personal information in your research record 
confidential but total confidentiality cannot be guaranteed. No information will be 
publicly accessible which could identify you as a participant. Research records will 
be stored and maintained in electronic form on NPS secure servers only 
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accessible by the researchers. Any hard copy material containing research 
findings, including a thesis, will not contain any personal information. 


Points of Contact. If you have any questions or comments about the research, or 
you experience an injury or have questions about any discomforts that you 
experience while taking part in this study, please contact the Principal Investigator, 
Dr John Osmundson, 831-656-3775, josmundson@nps.edu. Questions about your 
rights as a research subject, or any other concerns, may be addressed to the 
Naval Postgraduate School IRB Chair, CAPT John Schmidt, USN, 831-656-3864, 
jkschmid@nps.edu. 


Statement of Consent. | have read the information provided above. | have been 
given the opportunity to ask questions and all the questions have been answered 
to my satisfaction. | have been provided a copy of this form for my records and | 
agree to participate in this study. | understand that by agreeing to participate in 
this research and signing this form, | do not waive any of my legal rights. 
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Metric Feedback Instrument 

Please read the Software Project Management Effectiveness Metric provided to 
you. Then hypothetically consider using the previous survey questions on the 
next software project you work on and respond to the following questions. 


Manageability: A metric’s information should be available and concise. How 
manageable is the PME metric? (1 being unmanageable and 10 being easily 
manageable) 


1 2 3 4 5 6 7 8 9 10 

a ia Dl Pe ( 

Please provide any comments you may have on the metric’s 
manageability: 


Meaningful: A metric must be understandable and relevant to the recipient and 
provide a basis for decisions. How meaningful is the PME metric? (1 being not 
meaningful and 10 being very meaningful) 


1 2 3 4 5 6 7 8 9 10 

ee i ee ee ee ee ee ee 

Please provide any comments you may have on how meaningful the 
metric is: 


Actionable: Useful metrics information makes it clear what response is needed, 
as a compass makes it clear whether to turn left or right or stay on course. How 
actionable is the PME metric? (1 being not actionable at all and 10 being easily 
actionable) 


1 2 3 4 5 6 7 +8 9 10 
a 


Please provide any comments you may have on the metric’s actionability: 


Ambiguity: Information from metrics can have a number of meanings and may 
be misleading, of little use, or downright dangerous. How ambiguous is the PME 
metric? (1 being very ambiguous and 10 being completely unambiguous) 


1 2 3 4 5 6 7 +8 9 10 
2 ey ee 
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Please provide any comments you may have on the metric’s ambiguity: 


Reliability: The ability to trust the “instrument” is conditioned on the reliability of 
the measurement. How reliable is the PME metric? (1 being completely 
unreliable and 10 being completely reliable) 


1 2 3 4 5 6 7 +8 9 10 
Se ee 


Please provide any comments you may have on the metric’s reliability: 


Accuracy: A reasonable and known degree of a metric’s accuracy is essential. 
The compass showing north when we are going south can be fatal. How 
accurate is the PME metric? (1 being completely inaccurate and 10 being 
completely accurate) 


1 2 3 4 5 6 7 +8 9 10 
eS ee 


Please provide any comments you may have on the metric’s accuracy: 


Timely: Measures that warn of a disaster after it has happened are not useful. 
Consider measuring the PME of a project after the initial requirements 
development phase of a project. How timely is the PME metric? (1 being far too 
late and 10 being right on time) 


1 2 3 4 5 6 7 +8 9 10 
8 OS a i 


Please provide any comments you may have on the metric’s timeliness: 


Predictive: Some metrics information will signal impending problems much as a 
drop in oil pressure is the harbinger of engine failure. Consider measuring the 
PME of a project after the initial requirements development phase of a project. 
How predictive is the PME metric? (1 being completely non predictive and 10 
being very predictive) 


1 2 3 


4 5 6 7 8 9 10 
elt steel), lia, Je. El JES ea dee es 2s 
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Please provide any comments you may have on the metric’s predictability: 


Would you use, or would you like to see, the PME metric used on the next 
software project you work on? 


Yes No 
LT} 


Please briefly explain why or why not. 


Thank you for your participation! 
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APPENDIX F. SOFTWARE PROJECT MANAGEMENT 
EFFECTIVENESS METRIC REPORT CARD 


In a recent study on the effectiveness of software project management practices, 
you completed a survey on the project management practices in place on a 
project you have worked on. The data you provided has helped assist with the 
development of a software project management effectiveness (PME) metric. The 
research into the usefulness and the applicability of this metric is ongoing. The 
PME metric for your project was calculated from the responses in your survey. 
For your information, the results are presented here. 


Legend 
Area: The specific area of software project management measured. 
Score: Calculated rating from 0 to 10 (0 being the lowest and 10 being the - 
highest). : 
Average: Average score of projects in the study. 7 
_Explanation: Standard comments to explain the area score. 


Area Score Average Explanation 

Communication 49 6.9 A successful project requires constant and effective 
communication between project stakeholders. This 
score is an indication of the effectiveness of the 


communication in the project. 





Teamwork 6.1 7.0 As software is developed by teams, strong teamwork is 
essential to successfully completing a software project. 
This score is an indication of the effectiveness of the 
teamwork in the project. 








Leadership 4.3 7.1 In a software development environment leadership is 
how personnel in management positions exert social 
influence to enlist the aid and support of others in the 
accomplishment of project's goals. This score is an 
indication of the effectiveness of the leadership in the 
project. 


Organizational 7.1 7.1 Organizational commitment is the employee’s 
" psychological attachment to the organization and 


commitment organizational goals. This score is an indication of the 
effectiveness of the organizational commitment in the 
project. 

















Project 6.3 7.6 The project manager position is a key role in a software 
project’s organizational structure. This score is an 
Manager indication of the effectiveness of the project manager in 
the project. 





Stakeholder 44 6.6 Stakeholder engagement is concerned with the level of 
; involvement of all the different stakeholders during the 
Involvement project development effort. This score is an indication of 
the effectiveness of the stakeholder engagement in the 
project. 








Staffing and 6.1 6.5 Staffing and hiring is the ability to source human 

ae resources and put them in the right project role. This 

Hiring score is an indication of the effectiveness of the staffing 
and hiring in the project. 
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The people score is the average of the communication, 
teamwork, leadership, organizational commitment, 


project manager, stakeholder involvement, staffing and 
hiring scores. 






































Area Score Average Explanation 

Requirements 7.5 6.7 This process involves the management of the software 
requirements and is not to be confused with the 

management requirements development process. This score is an 
indication of the effectiveness of the requirements 
management in the project. 

Project 6.5 6.6 Project monitoring is the process of keeping the project, 

project related factors and project metrics under 

Monitoring and continuous observation. This score is an indication of the 

Control effectiveness of the project monitoring and control in the 
project. 

Project planning and estimation is the process of 
Project 6.7 6.6 establishing and maintaining the plans that define the 
planning and project’s work activities. This score is an indication of the 
estimation effectiveness of the project planning and estimation in 

the project. 
Scope 6.5 5.9 This is the process of defining the scope of the project 
and keeping track of any changes to the scope. This 
management score is an indication of the effectiveness of the scope 
management in the project. 


Process : This score is the average of the requirements 


management, project monitoring and control, project 
planning and estimation and scope management scores. 








Area Score Average Explanation 


Configuration 10 6.3 Software configuration management is the discipline that 
enables us to keep evolving software products under 
management control, and thus contributes to satisfying quality 


constraints. This score is an_ indication of the 
effectiveness of the configuration management in the 


project. 





| : 7.1 Quality engineering involves all activities put in place to 
Qua ty : 9.5 ensure the development of a high quality product. This 
engineering score is an indication of the effectiveness of the quality 








engineering in the project. 





Product 9.7 6.7 The product score is the average of the configuration 
; management and quality engineering scores. 

















Area Score Average Explanation 

Risk 6.0 6.0 Risk assessment involves risk identification, risk analysis 
and risk prioritization. This score is an indication of the 

assessment effectiveness of the risk assessment in the project. 

Risk control 6.1 56 Risk control involves risk management planning, risk 
resolution, and risk monitoring. This score is an 
indication of the effectiveness of the risk control in the 
project. 





Risk management is an inherent aspect of any software 


project. This score is the average of the risk assessment 
and control scores. 
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PME Score The project management effectiveness score is the 
weighted sum of the people, process, product and 
risk scores. It gives an_ indication of the 


effectiveness of the software project management in 
the project. 
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APPENDIX G. FEEDBACK METRIC INSTRUMENT RESPONSES 


Score Manageability Comments 


10 It's a short list easily manageable and areas and overall scores can be 
clearly be identified. 


While getting back the "score" was interesting, not having any 
explanation of why or how the score was derived limits 


8 | am not sure what you mean by manageable 


Toro) c= Mam (=r-lallarepiellatsr-t-m Oxo) anlantcy alc 


As a high-level indicator, | think it clearly defines the areas of good 
performance and the areas of concern. I'm not quite sure whether we 
can treat these metrics as values, and while | notice that the final score 
was a weighted sum, it's not quite apparent which areas are given the 
most weighting, especially since the unweighted average came out so 
close to the final score. Realistically, | am not going to be able to address 
each of the low scoring areas simultaneously, so if | have to pick an area 
of improvement | want to pick the one that's going to give me the best 
chance of improving my project success, and that may not be the one 
with the lowest score. While knowing the average score is probably a 
great metric for you, I'm not sure what | can do with it. | would like to see 
target bands or something similar that that gives me a clearly defined 
goal to aim for. I'm not too concerned about beating the average 
(commercial world might look at this differently, i.e., better than average 
makes them look good when to win a contract), | want to know how a 
certain score will impact my project. Perhaps you can use the data from 
your surveys to work out some bands, it’s entirely subjective, but in 
reality so was the survey. 


Without the average, these metrics are meaningless, so the point is how 
significant and precise is the average ? 
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The scores are clear and simple; you can easily assess how well the 
Surveyed group performed and how well you did. The areas are 
straightforward along with their explanations. However, in order to begin 
self improvement it would be good to see a breakdown of key techniques 
in each area and how you scored on each if it's available. That you could 
begin focusing on some techniques that you were lacking in. Though this 
extra information may work adversely against the manageability of the 
report which is really concise at the moment. 

For some of my scores, | Know enough that | agree - good score for CM, 
not so good for risk management. But I'm not sure why | got a poor score 
for Leadership. 


10 =2ONil 


Score Actionability Comments 


7 As before, the areas where we need to improve are clear, but its hard to 
prioritize which ones to go after first. 


7 When a project is performing very poorly by a rather large gap, the 
metric provides clear insight; in other cases it will be less clear what 
action to take. 


| touched on this a bit in the last question, | think for some areas that are 
a bit broader - what you're doing well and what you need to improve on 
may not be so clear. But for the areas that are based on a technique, like 
‘Configuration Management’, which is one | need to improve on, | can 
clearly see how to improve that. And coincidently my company it 
currently working hard to improve our standard of configuration 
management. 


Score Ambiguity 


| don’t think the PME metric is ambiguous but | think it is still lacking in 
definition. It tells me how | scored compared to the average, but it does 
not tell me if the scores are good or not. My projects final PME score 
was higher than average, but | Know we performed atrociously, so am | 
to assume that because we beat the average we are actually doing 
alright? What projects did you use to create your average, did you get a 
good spread of projects or just a whole bunch of bad projects? Did we 
only beat the average because most of the projects sampled were bad 
projects? 
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The generated report is unambiguous, but some of the questions in the 
original survey could have a wide variance in interpretation. 


Examples would remove ambiguities in certain cases 

The areas create clear boundaries and the scores that relate to them are 
simple to understand. | think there's only a little and maybe not even 
crucial ambiguity, like | stated in the earlier questions, where within a 
broad area such as ‘Communication’ you may not know what you are 
doing right and what you need to improve on within that area. 


Score Reliability Comments 


9 | am convinced that the scores from the survey will produce reliable 
metrics. The problem comes from getting a reliable source of data to 
produce the metrics 

7 Nil 


Score Accuracy Comments 


here is an inherent inaccuracy that comes from doing something 
subjective like a survey and that comes from the bias of the person 
completing the survey. This could probably be combated by surveying 
a wide sample of people from the same project and getting someone 
independent to make an assessment 


6 Nil 
7 For some cases the scale of assessment could have been a 
percentage rather than a five level scale 


9 | found it very accurate. The areas | consider myself good at or 
lacking were reflected spot on in the report. 
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Score timeliness Comments 


| disagree with the statement that measures warning of disaster after it 
has happened are not useful. In my opinion it depends entirely on how 
you intend to use the measures. Back on topic, this metric would be 
considered timely if produced just after the initial requirements 
development phase of a project, but | would not be using these metrics 
as a predictive tool. 


| think you'd need that real grasp of what work is to be carried out and 
the risks involved with getting it to the requirements first. So | think this is 
a good time. 


Would you know enough to get meaningful answers to the questions? 
Just because a good CM infrastructure was in place, suppose 
developers stopped using CM and checked in code updates less and 
less often? 


Score Predictive Comments 


While this metric could possibly be predictive, | would not use this metric 
as a predictive tool. | would be using these metrics at the end of a project 
as a way of assessing how we did in the project and identifying what 
areas we need to improve for our next project. | think there are better 
predictive measures out there, PSM has made a successful business out 
of determining the best measures to use at different stages of a project. 
As far as | know, organizations make money out of running multiple 
software projects, they are not normally interested in starting up a 
company to complete one project before shutting down again 

For some areas, low scores will be very predictive of problems (e.g., 
Stakeholder involvement). Others | would give less weight to that early 
on in the project (e.g., Teamwork). 


9 Could be proactive if the PM is confronted to higher risks in the 
upcoming projects 


8 This is a tough one. It reinforces a lot of management techniques that 
need to be considered at the requirements stage of development and 
has the chance to improve predicting aspects of the project. 
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Yes/No Please briefly explain why or why not 


This is the sort of metric you need to use at the end of each project. It 
clearly identifies strengths and weakness, and from that you can select 
areas that need improvement. You can compare previous results with 
later results to see if the processes you have introduced have actually 
resulted in improvements to those weaker areas. | think you have 
created a great tool to analyze strengths and weaknesses of an 
project/organization but your metrics report probably needs a bit of 
work before you could use it as a predictive management tool. This tool 
is perfect for use at the end of a project because you can use it to 
identify areas that need to improve to make your next project work 
better. 
No, not as is. The original questions need to be less open to 
interpretation. | would likely begin with a metric reduced in scope to 
questions that could be answered objectively, then as the team is 
allowed time to build trust and establish relationships, introduce the 
more subjective question areas 

1 At least | can evaluate whether the process is evolving or not. and also 
helps to fix problems per domain 


: Yes | found the survey very useful as a chance to reflect on my last 
project and I've found it's made me think about my current projects. 


Since not feedback was given on how survey answers were converted 
into PME metrics, it is not possible for me to understand what 
behaviors were good and which behaviors need to be changed. Also, 
as for the leadership questions, senior leadership is not something that 
| or a typical project leader can really influence, control, or change. 


1 Yes, if the metrics is available throughout the projects 
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